Using prediction information with light fields

ABSTRACT

A computer-implemented method and system may include receiving, by a computing device associated with an image processor, a sensor data associated with a user device; generating, by the computing device, light field data based on a predicted behavior determined at least in part by the received sensor data; and communicating, by the computing device, the light field data to the user device.

FIELD

The present disclosure relates generally to communication of visualcontent to a user device, and more specifically to communication ofvisual content in response to an action in association with the userdevice.

BACKGROUND

Interactive systems that include visual content are often implementedwith some image processing functions performed remotely from the userdevice. As a result, for real time applications there is a lag in thedelivery of continuous visual content (e.g., streaming of visual contentwhere the scene is changing) or in response to user device inputs (e.g.,a scene change due to an input received from the user device, such asbetween a data input resulting from a user response to the presentationof a first visual content and the presentation of a second visualcontent that is responsive to the data input). This lag may be due tocommunication time between the user device and the remote imageprocessing facility, the image processing time for generating the secondvisual content, and the like. This lag can cause unintended consequenceswith respect to a user experience, such as where the user consciouslysenses the lag (e.g., the user sensing that the image content update isslow or discontinuous) or unconsciously senses the lag (e.g., causingmotion sickness). There is a need for methods and systems that reducethe perceived lag between consecutive displays of visual content in avisual-based interactive system.

SUMMARY

In an aspect, a method may include communicating, by a computing device,a light field data to a user device, wherein the light field datacomprises a visual content greater than a display field of view of theuser device. In embodiments, the visual content greater than a displayfield of view of the user device may be at the time of communication ofthe light field data. The visual content greater than a display field ofview of the user device may be at the expected time of receipt of thelight field data. The light field data may include a three-dimensionalvolume describing the light flowing in a plurality of directions througha plurality of light field data points. The light field data may becommunicated to the user device for display as virtual reality,augmented reality, or mixed reality content. The processor adapted toprocess light field data may generate display data from the light fielddata. The user device may be communicatively coupled to a display forpresentation of the display data. The light field data may include athree-dimensional volume of light field data points. Thethree-dimensional volume may be omnidirectional with respect to the userdevice as an origin. The three-dimensional volume may be a partialvolume with respect to the user device as an origin. The partial volumemay have a field of view characteristic. The three-dimensional volumemay be a partial volume with respect to a determined direction of motionof the user device. The light field data may be based on a data modelfor light characterization in a volume of space.

In an aspect, a system may include a computing device associated with animage processor, the computing device configured to store a set ofinstructions that, when executed, cause the computing device tocommunicate a light field data to a user device, the user devicecomprising a processor adapted to process light field data, wherein thelight field data comprises a visual content greater than a display fieldof view of the user device. In embodiments, the visual content greaterthan a display field of view of the user device may be at the time ofcommunication of the light field data. The visual content greater than adisplay field of view of the user device may be at the expected time ofreceipt of the light field data. The light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points. The lightfield data may be communicated to the user device for display as virtualreality, augmented reality, or mixed reality content. The processoradapted to process light field data may generate display data from thelight field data. The user device may be communicatively coupled to adisplay for presentation of the display data. The light field data mayinclude a three-dimensional volume of light field data points. Thethree-dimensional volume may be omnidirectional with respect to the userdevice as an origin. The three-dimensional volume may be a partialvolume with respect to the user device as an origin. The partial volumemay have a field of view characteristic. The three-dimensional volumemay be a partial volume with respect to a determined direction of motionof the user device. The light field data may be based on a data modelfor light characterization in a volume of space.

In an aspect, a method may include communicating, by a computing device,a first light field data to a user device, the user device comprising aprocessor adapted to process light field data; receiving, by thecomputing device, a sensor data associated with the user device; andcommunicating, by the computing device, a second light field data, basedat least in part on the sensor data, to the user device. In embodiments,the light field data may include a three-dimensional volume describingthe light flowing in a plurality of directions through a plurality oflight field data points. The first light field data may include a firstsurface volume in a surrounding environment with respect to the userdevice, and the second light field data may include a second surfacevolume in a surrounding environment with respect to the user device. Thelight field data may be communicated to the user device for display asvirtual reality, augmented reality, or mixed reality content. The sensordata may represent a change in a viewing perspective with respect to thevirtual reality, augmented reality, or mixed reality content. The userdevice may be a head-mounted device and the sensor data may represent amotion of the head-mounted device. The user device may be a wearabledevice and the sensor data may represent a motion of the wearabledevice. The user device may be a hand-held device and the sensor datamay represent a motion of the hand-held device. The sensor data mayfurther represent a motion of a head-mounted device. The processoradapted to process light field data may generate display data from thelight field data. The user device may be communicatively coupled to adisplay for presentation of the display data. The sensor data mayinclude a position sensor data or motion sensor data. The light fielddata may include a three-dimensional volume of light field data points.The three-dimensional volume may change from the first light field datato the second light field data. The three-dimensional volume may beomnidirectional with respect to the user device as an origin. Thethree-dimensional volume may be a partial volume with respect to theuser device as an origin. The partial volume may have a field of viewcharacteristic. The three-dimensional volume may be a partial volumewith respect to a determined direction of motion of the user device. Thefirst light field data may include a first light field spatial volume,and the second light field data may include a second light field spatialvolume that is adjusted in size based on the received sensor data. Thelight field data may be based on a data model for light characterizationin a volume of space.

In an aspect, a system may include a computing device associated with animage processor, the computing device configured to store a set ofinstructions that, when executed, cause the computing device to:communicate a first light field data to a user device, the user devicecomprising a processor adapted to process light field data; receive asensor data associated with the user device; and communicate a secondlight field data, based at least in part on the sensor data, to the userdevice. In embodiments, the light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points. The firstlight field data may include a first surface volume in a surroundingenvironment with respect to the user device, and the second light fielddata may include a second surface volume in a surrounding environmentwith respect to the user device. The light field data may becommunicated to the user device for display as virtual reality,augmented reality, or mixed reality content. The sensor data mayrepresent a change in a viewing perspective with respect to the virtualreality, augmented reality, or mixed reality content. The user devicemay be a head-mounted device and the sensor data may represent a motionof the head-mounted device. The user device may be a wearable device andthe sensor data may represent a motion of the wearable device. The userdevice may be a hand-held device and the sensor data may represent amotion of the hand-held device. The sensor data may further represent amotion of a head-mounted device. The processor adapted to process lightfield data may generate display data from the light field data. The userdevice may be communicatively coupled to a display for presentation ofthe display data. The sensor data may include a position sensor data ormotion sensor data. The light field data may include a three-dimensionalvolume of light field data points. The three-dimensional volume maychange from the first light field data to the second light field data.The three-dimensional volume may be omnidirectional with respect to theuser device as an origin. The three-dimensional volume may be a partialvolume with respect to the user device as an origin. The partial volumemay have a field of view characteristic. The three-dimensional volumemay be a partial volume with respect to a determined direction of motionof the user device. The first light field data may include a first lightfield spatial volume, and the second light field data may include asecond light field spatial volume that is adjusted in size based on thereceived sensor data. The light field data may be based on a data modelfor light characterization in a volume of space.

In aspect, a method may include receiving at a computing device a sensordata associated with a user device, the user device comprising aprocessor adapted to process light field data; and communicating lightfield data, based at least in part on the sensor data, to the userdevice. In embodiments, the light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points. The lightfield data may be communicated to the user device for display as virtualreality, augmented reality, or mixed reality content. The sensor datamay represent a viewing perspective with respect to the virtual reality,augmented reality, or mixed reality content. The user device may be ahead-mounted device and the sensor data may represent a motion of thehead-mounted device. The user device may be a wearable device and thesensor data may represent a motion of the wearable device. The userdevice may be a hand-held device and the sensor data may represent amotion of the hand-held device. The sensor data may further represent amotion of a head-mounted device. The processor adapted to process lightfield data may generate display data from the light field data. The userdevice may be communicatively coupled to a display for presentation ofthe display data. The sensor data may include position sensor data. Thesensor data may include motion sensor data. The light field data mayinclude a three-dimensional volume of light field data points. Thethree-dimensional volume may be omnidirectional with respect to the userdevice as an origin. The three-dimensional volume may be a partialvolume with respect to the user device as an origin. The partial volumemay have a field of view characteristic. The three-dimensional volumemay be a partial volume with respect to a determined direction of motionof the user device. The light field data may be based on a data modelfor light characterization in a volume of space.

In an aspect, a system may include a computing device associated with animage processor, the computing device configured to store a set ofinstructions that, when executed, cause the computing device to: receivea sensor data associated with a user device, the user device comprisinga processor adapted to process light field data; and communicate lightfield data, based at least in part on the sensor data, to the userdevice. In embodiments, the light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points. The lightfield data may include a three-dimensional volume describing the lightflowing in a plurality of directions through a plurality of light fielddata points. The light field data may be communicated to the user devicefor display as virtual reality, augmented reality, or mixed realitycontent. The sensor data may represent a viewing perspective withrespect to the virtual reality, augmented reality, or mixed realitycontent. The user device may be a head-mounted device and the sensordata may represent a motion of the head-mounted device. The user devicemay be a wearable device and the sensor data may represent a motion ofthe wearable device. The user device may be a hand-held device and thesensor data may represent a motion of the hand-held device. The sensordata may further represent a motion of a head-mounted device. Theprocessor adapted to process light field data may generate display datafrom the light field data. The user device may be communicativelycoupled to a display for presentation of the display data. The sensordata may include position sensor data. The sensor data may includemotion sensor data. The light field data may include a three-dimensionalvolume of light field data points. The three-dimensional volume may beomnidirectional with respect to the user device as an origin. Thethree-dimensional volume may be a partial volume with respect to theuser device as an origin. The partial volume may have a field of viewcharacteristic. The three-dimensional volume may be a partial volumewith respect to a determined direction of motion of the user device. Thelight field data may be based on a data model for light characterizationin a volume of space.

In an aspect, a method may include receiving a light field data by auser device, wherein the light field data comprises a visual contentgreater than a display field of view of the user device. In embodiments,the light field data may include a three-dimensional volume describingthe light flowing in a plurality of directions through a plurality oflight field data points. The light field data may be displayed on theuser device for display as virtual reality, augmented reality, or mixedreality content. The user device may be adapted to process light fielddata to generate display data from the light field data. The user devicemay be communicatively coupled to a display for presentation of thedisplay data. The light field data may include a three-dimensionalvolume of light field data points. The light field data may be based ona data model for light characterization in a volume of space.

In an aspect, a system may include a user device, the user deviceconfigured to store a set of instructions that, when executed, cause theuser device to receive a light field data, wherein the light field datamay include a visual content greater than a display field of view of theuser device. In embodiments, the light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points. The lightfield data may be displayed on the user device for display as virtualreality, augmented reality, or mixed reality content. The user devicemay be adapted to process light field data to generate display data fromthe light field data. The user device may be communicatively coupled toa display for presentation of the display data. The light field data mayinclude a three-dimensional volume of light field data points. The lightfield data may be based on a data model for light characterization in avolume of space.

In an aspect, a method may include receiving a first light field data bya user device, the user device comprising a processor adapted to processlight field data; communicating a sensor data associated with the userdevice; and receiving a second light field data, based at least in parton the sensor data, by the user device. In embodiments, the light fielddata may include a three-dimensional volume describing the light flowingin a plurality of directions through a plurality of light field datapoints. The light field data may be displayed by the user device asvirtual reality, augmented reality, or mixed reality content. The sensordata may include a position sensor data or motion sensor data. The firstlight field data may include a first light field spatial volume, and thesecond light field data may include a second light field spatial volumethat is adjusted in size based on the communicated sensor data.

In an aspect, a system may include a user device, the user deviceconfigured to store a set of instructions that, when executed, cause theuser device to receive a first light field data by a user device, theuser device comprising a processor adapted to process light field data;communicate a sensor data associated with the user device; and receive asecond light field data, based at least in part on the sensor data, bythe user device. In embodiments, the light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points. The lightfield data may be displayed by the user device as virtual reality,augmented reality, or mixed reality content. The sensor data may includea position sensor data or motion sensor data. The first light field datamay include a first light field spatial volume, and the second lightfield data may include a second light field spatial volume that isadjusted in size based on the communicated sensor data.

In an aspect, a method may include communicating a sensor data by a userdevice; and receiving light field data, based at least in part on thesensor data, by the user device. In embodiments, the light field datamay include a three-dimensional volume describing the light flowing in aplurality of directions through a plurality of light field data points.The light field data may be displayed by the user device as virtualreality, augmented reality, or mixed reality content. The user devicemay include a head-mounted device and the sensor data may represent amotion of the head-mounted device. The user device may include awearable device and the sensor data may represent a motion of thewearable device. The user device may include a hand-held device and thesensor data may represent a motion of the hand-held device. The userdevice may be adapted to process light field data to generate displaydata from the light field data. The sensor data may include positionsensor data. The sensor data may include motion sensor data. The lightfield data may include a three-dimensional volume of light field datapoints. The three-dimensional volume may be a partial volume withrespect to a determined direction of motion of the user device. Thelight field data may be based on a data model for light characterizationin a volume of space.

In an aspect, a system may include a user device, the user deviceconfigured to store a set of instructions that, when executed, cause theuser device to communicate a sensor data by a user device; and receivelight field data, based at least in part on the sensor data, by the userdevice. In embodiments, the light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points. The lightfield data may be displayed by the user device as virtual reality,augmented reality, or mixed reality content. The user device may includea head-mounted device and the sensor data may represent a motion of thehead-mounted device. The user device may include a wearable device andthe sensor data may represent a motion of the wearable device. The userdevice may include a hand-held device and the sensor data may representa motion of the hand-held device. The user device may be adapted toprocess light field data to generate display data from the light fielddata. The sensor data may include position sensor data. The sensor datamay include motion sensor data. The light field data may include athree-dimensional volume of light field data points. Thethree-dimensional volume may be a partial volume with respect to adetermined direction of motion of the user device. The light field datamay be based on a data model for light characterization in a volume ofspace.

In an aspect, a method may include communicating a sensor dataassociated with a user device, the user device comprising a processoradapted to process light field data, and receiving light field data,based at least in part on the sensor data, by the user device. Inembodiments, the light field data may include a three-dimensional volumedescribing the light flowing in a plurality of directions through aplurality of light field data points.

In an aspect, a system may include a user device configured to store aset of instructions that, when executed, cause the user device tocommunicate a sensor data associated with a user device, the user devicecomprising a processor adapted to process light field data, and receivelight field data, based at least in part on the sensor data, by the userdevice. In embodiments, the light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points.

In an aspect, a method for a user device may include receiving, from acomputing device, a light field data generated by the computing device,the light field data representing a portion of a viewed scene by theuser device, wherein the light field data comprises a three-dimensionalvolume describing the light flowing in a plurality of directions througha plurality of light field data points. In embodiments, the user devicemay be configured to display augmented reality or mixed reality contentwhere the portion of the viewed scene is presented against a surroundingenvironment viewed on the user device. The portion of the viewed scenemay remain fixed with reference to the surrounding environment. Theportion of the viewed scene may be oriented with respect to an object.The object may be a virtual object presented against the surroundingenvironment as viewed through the user device. The object may be a realobject in the surrounding environment as viewed through the user device.The light field data may be generated with respect to a resolutionconstraint. The resolution constraint may be with respect to a number oflight field vectors for each of the plurality of light field datapoints. The resolution constraint may be with respect to a densityfunction of the plurality of light field data points. The densityfunction may be with respect to a field of view of the portion of theviewed scene. The resolution constraint may be with respect to adirection function of light field vectors within the portion of theviewed scene. The light field data may be generated with low resolutionlight fields and high resolution light fields representing the portionof the viewed scene. The user device may be communicatively coupled to adisplay for presentation of display data.

In an aspect, a system may include a user device, the user deviceconfigured to store a set of instructions that, when executed, cause theuser device to receive from a computing device a light field datagenerated by the computing device, the light field data representing aportion of a viewed scene by the user device, wherein the light fielddata comprises a three-dimensional volume describing the light flowingin a plurality of directions through a plurality of light field datapoints. In embodiments, the user device may be configured to displayaugmented reality or mixed reality content where the portion of theviewed scene is presented against a surrounding environment viewed onthe user device. The portion of the viewed scene may remain fixed withreference to the surrounding environment. The portion of the viewedscene may be oriented with respect to an object. The object may be avirtual object presented against the surrounding environment as viewedthrough the user device. The object may be a real object in thesurrounding environment as viewed through the user device. The lightfield data may be generated with respect to a resolution constraint. Theresolution constraint may be with respect to a number of light fieldvectors for each of the plurality of light field data points. Theresolution constraint may be with respect to a density function of theplurality of light field data points. The density function may be withrespect to a field of view of the portion of the viewed scene. Theresolution constraint may be with respect to a direction function oflight field vectors within the portion of the viewed scene. The lightfield data may be generated with low resolution light fields and highresolution light fields. The user device may be communicatively coupledto a display for presentation of display data.

In an aspect, a method may include receiving, from a computing device, alight field data representing a portion of a viewed scene by a userdevice, wherein the portion of the viewed scene describes alocation-based portion of a three-dimensional volume in a surroundingenvironment of the user device. In embodiments, the light field data mayinclude a three-dimensional volume describing the light flowing in aplurality of directions through a plurality of light field data points.The user device may be configured to present augmented reality or mixedreality content where the portion of the viewed scene is displayedagainst the surrounding environment viewed on the user device. Theportion of the viewed scene may remain fixed with reference to thesurrounding environment. The portion of the viewed scene may be orientedwith respect to an object. The object may be a virtual object presentedagainst the surrounding environment as viewed on the user device. Theobject may be a real object in the surrounding environment as viewed onthe user device. The light field data may be generated with respect to aresolution constraint. The resolution constraint may be with respect toa number of light field vectors for each of a plurality of light fielddata points. The resolution constraint may be with respect to a densityfunction of a plurality of light field points. The density function maybe with respect to a field of view of the portion of the viewed scene.The resolution constraint may be with respect to a direction function oflight field vectors within the portion of the viewed scene. The lightfield data may be generated with low resolution light fields and highresolution light fields. The user device may be communicatively coupledto a display for presentation of display data.

In an aspect, a system may include a user device, the user deviceconfigured to store a set of instructions that, when executed, cause theuser device to receive from a computing device a light field datarepresenting a portion of a viewed scene by a user device, wherein theportion of the viewed scene describes a location-based portion of athree-dimensional volume in a surrounding environment of the userdevice. In embodiments, the light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points. The userdevice may be configured to present augmented reality or mixed realitycontent where the portion of the viewed scene is displayed against thesurrounding environment viewed on the user device. The portion of theviewed scene may remain fixed with reference to the surroundingenvironment. The portion of the viewed scene may be oriented withrespect to an object. The object may be a virtual object presentedagainst the surrounding environment as viewed on the user device. Theobject may be a real object in the surrounding environment as viewed onthe user device. The light field data may be generated with respect to aresolution constraint. The resolution constraint may be with respect toa number of light field vectors for each of a plurality of light fielddata points. The resolution constraint may be with respect to a densityfunction of a plurality of light field points. The density function maybe with respect to a field of view of the portion of the viewed scene.The resolution constraint may be with respect to a direction function oflight field vectors within the portion of the viewed scene. The lightfield data may be generated with low resolution light fields and highresolution light fields. The user device may be communicatively coupledto a display for presentation of display data.

In an aspect, a method may include generating, by a computing device, alight field data, wherein the light field data comprises athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points; andcommunicating, by the computing device, the light field data andlocation information to the user device, a user device comprising aprocessor adapted to process light field data, wherein the locationinformation describes a location-based portion of a three-dimensionalvolume in a surrounding environment of the user device. In embodiments,the user device may be configured to present augmented reality or mixedreality content where the location information identifies a region ofthe surrounding environment for processing of display data as presentedagainst the surrounding environment viewed by the user device. Theregion of the surrounding environment may remain fixed with reference tothe surrounding environment. The region of the surrounding environmentmay remain fixed with reference to the user device. A region of thesurrounding environment may be oriented with respect to an object. Theobject may be a virtual object presented against the surroundingenvironment as viewed through the user device. The object may be a realobject in the surrounding environment as viewed through the user device.The light field data may be generated with respect to a resolutionconstraint. The resolution constraint may be with respect to a number oflight field vectors for each of the plurality of light field datapoints. The resolution constraint may be with respect to a densityfunction of the plurality of light field points. The density functionmay be with respect to a field of view of the portion of a viewed scene.The resolution constraint may be with respect to a direction function oflight field vectors within the portion of a viewed scene. The lightfield data may be generated with resolution stitches with low resolutionlight fields and high resolution light fields. The user device may becommunicatively coupled to a display for presentation of display data.

In an aspect, a system may include a computing device associated with animage processor, the computing device configured to store a set ofinstructions that, when executed, cause the computing device to generatea light field data, wherein the light field data comprises athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points; andcommunicate the light field data and location information to the userdevice, a user device comprising a processor adapted to process lightfield data, wherein the location information describes a location-basedportion of a three-dimensional volume in a surrounding environment ofthe user device. In embodiments, the user device may be configured topresent augmented reality or mixed reality content where the locationinformation identifies a region of the surrounding environment forprocessing of display data as presented against the surroundingenvironment viewed by the user device. The region of the surroundingenvironment may remain fixed with reference to the surroundingenvironment. The region of the surrounding environment may remain fixedwith reference to the user device. A region of the surroundingenvironment may be oriented with respect to an object. The object may bea virtual object presented against the surrounding environment as viewedthrough the user device. The object may be a real object in thesurrounding environment as viewed through the user device. The lightfield data may be generated with respect to a resolution constraint. Theresolution constraint may be with respect to a number of light fieldvectors for each of the plurality of light field data points. Theresolution constraint may be with respect to a density function of theplurality of light field points. The density function may be withrespect to a field of view of the portion of a viewed scene. Theresolution constraint may be with respect to a direction function oflight field vectors within the portion of a viewed scene. The lightfield data may be generated with resolution stitches with low resolutionlight fields and high resolution light fields. The user device may becommunicatively coupled to a display for presentation of display data.

In an aspect, a method may include communicating to a user device alight field data and location information, the user device comprising aprocessor adapted to process the light field data; and processing, bythe user device, the light field data to represent a portion of a viewedscene by the user device based on the location information, wherein theportion of the viewed scene describes a location-based portion of athree-dimensional volume in a surrounding environment of the userdevice. In embodiments, the light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points. The userdevice may be configured to present augmented reality or mixed realitycontent where the portion of the viewed scene is presented against thesurrounding environment viewed by the user device. The portion of theviewed scene may remain fixed with reference to the surroundingenvironment. The portion of the viewed scene may be oriented withrespect to an object. The object may be a virtual object presentedagainst the surrounding environment as viewed through the user device.The object may be a real object in the surrounding environment as viewedthrough the user device. The light field data may be generated withrespect to a resolution constraint. The resolution constraint may bewith respect to a number of light field vectors for each of a pluralityof light field data points. The resolution constraint may be withrespect to a density function of a plurality of light field data points.The density function may be with respect to a field of view of theportion of the viewed scene. The resolution constraint may be withrespect to a direction function of light field vectors within theportion of the viewed scene. The light field data may be generated withresolution stitches with low resolution light fields and high resolutionlight fields representing the portion of the viewed scene. The userdevice may be communicatively coupled to a display for presentation ofdisplay data.

In an aspect, a system may include a computing device associated with animage processor, the computing device configured to store a set ofinstructions that, when executed, cause the computing device tocommunicate to a user device a light field data and locationinformation, the user device comprising a processor adapted to processthe light field data; and process the light field data to represent aportion of a viewed scene by the user device based on the locationinformation, wherein the portion of the viewed scene describes alocation-based portion of a three-dimensional volume in a surroundingenvironment of the user device. In embodiments, the light field data mayinclude a three-dimensional volume describing the light flowing in aplurality of directions through a plurality of light field data points.The user device may be configured to present augmented reality or mixedreality content where the portion of the viewed scene is presentedagainst the surrounding environment viewed by the user device. Theportion of the viewed scene may remain fixed with reference to thesurrounding environment. The portion of the viewed scene may be orientedwith respect to an object. The object may be a virtual object presentedagainst the surrounding environment as viewed through the user device.The object may be a real object in the surrounding environment as viewedthrough the user device. The light field data may be generated withrespect to a resolution constraint. The resolution constraint may bewith respect to a number of light field vectors for each of a pluralityof light field data points. The resolution constraint may be withrespect to a density function of a plurality of light field data points.The density function may be with respect to a field of view of theportion of the viewed scene. The resolution constraint may be withrespect to a direction function of light field vectors within theportion of the viewed scene. The light field data may be generated withresolution stitches with low resolution light fields and high resolutionlight fields representing the portion of the viewed scene. The userdevice may be communicatively coupled to a display for presentation ofdisplay data.

In an aspect, a method may include communicating, by a computing device,a first light field data to a user device; receiving, by the computingdevice, a sensor data associated with the user device; predicting abehavior based at least in part from the received sensor data;generating with the computing device a second light field data based atleast in part on the predicted behavior; and communicating, by thecomputing device, the second light field data to the user device. Inembodiments, the behavior may be a behavior of the user device. Thebehavior of the user device may be determined by the computing device tobe representative of a user behavior. The light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points. The firstlight field data may include a first surface volume in a surroundingenvironment with respect to the user device, and the second light fielddata comprises a second surface volumes in a surrounding environmentwith respect to the user device. The predicted behavior may be apredicted future position. The sensor data may represent the predictedfuture position based on a velocity vector or acceleration vectordetermined from the sensor data. The second light field data may beassociated with a different location within a surrounding environmentwith respect to the first light field data. The predicted behavior maybe a predicted viewing angle. The predicted viewing angle may be basedon a line of sight vector determined from the sensor data. The secondlight field data may be associated with a different location within asurrounding environment with respect to the first light field data. Thepredicted behavior may be based on a prior action. The prior action maybe determined to have similar behavior characteristics to a presentaction. The predicted behavior may be based on an action of a seconduser device. The second user device may be determined to be in asurrounding environment through the sensor data. The first light fielddata may include a volume of space proximate to the user device and thevolume of space associated with the second light field data may bedetermined by the predicted behavior. The first light field data mayinclude a light field resolution and the light field resolutionassociated with the second light field data may be determined by thepredicted behavior. The first light field data may include adirectionality with respect to a user device perspective and thedirectionality associated with the second light field data may bedetermined by the predicted behavior. The directionality of the firstlight field data may be associated with a field of view with respect toa user device perspective and the field of view associated with thesecond light field data may be determined by the predicted behavior. Thefirst light field data may include a light field point density and thelight field point density associated with the second light field datamay be determined by the predicted behavior. The first light field datamay be associated with a first head position of the user device and thehead position associated with the second light field data may bedetermined by the predicted behavior. The head position associated withthe second light field may be at least in part determined by aprobability gradient of possible head positions based on a previous headposition. The predicted behavior may be determined at least in partthrough machine learning associated with previous user behavior. Thebehavior may be determined to be a user contact with an object, whereinthe second light field data may be located to be concurrent with alocation of the object. The object may be a physical object in asurrounding environment. The object may be a virtual object insurrounding virtual environment. The user device may be communicativelycoupled to a display for presentation of display data. The sensor datamay include position sensor data. The sensor data may include motionsensor data.

In an aspect, a system may include a computing device associated with animage processor, the computing device configured to store a set ofinstructions that, when executed, cause the computing device tocommunicate, by a computing device, a first light field data to a userdevice; receiving, by the computing device, a sensor data associatedwith the user device; predict a behavior based at least in part from thereceived sensor data; generate with the computing device a second lightfield data based at least in part on the predicted behavior; andcommunicate, by the computing device, the second light field data to theuser device. In embodiments, the behavior may be a behavior of the userdevice. The behavior of the user device may be determined by thecomputing device to be representative of a user behavior. The lightfield data may include a three-dimensional volume describing the lightflowing in a plurality of directions through a plurality of light fielddata points. The first light field data may include a first surfacevolume in a surrounding environment with respect to the user device, andthe second light field data comprises a second surface volume in asurrounding environment with respect to the user device. The predictedbehavior may be a predicted future position. The sensor data mayrepresent the predicted future position based on a velocity vector oracceleration vector determined from the sensor data. The second lightfield data may be associated with a different location within asurrounding environment with respect to the first light field data. Thepredicted behavior may be a predicted viewing angle. The predictedviewing angle may be based on a line of sight vector determined from thesensor data. The second light field data may be associated with adifferent location within a surrounding environment with respect to thefirst light field data. The predicted behavior may be based on a prioraction. The prior action may be determined to have similar behaviorcharacteristics to a present action. The predicted behavior may be basedon an action of a second user device. The second user device may bedetermined to be in a surrounding environment through the sensor data.The first light field data may include a volume of space proximate tothe user device and the volume of space associated with the second lightfield data may be determined by the predicted behavior. The first lightfield data may include a light field resolution and the light fieldresolution associated with the second light field data may be determinedby the predicted behavior. The first light field data may include adirectionality with respect to a user device perspective and thedirectionality associated with the second light field data may bedetermined by the predicted behavior. The directionality of the firstlight field data may be associated with a field of view with respect toa user device perspective and the field of view associated with thesecond light field data may be determined by the predicted behavior. Thefirst light field data may include a light field point density and thelight field point density associated with the second light field datamay be determined by the predicted behavior. The first light field datamay be associated with a first head position of the user device and thehead position associated with the second light field data may bedetermined by the predicted behavior. The head position associated withthe second light field may be at least in part determined by aprobability gradient of possible head positions based on a previous headposition. The predicted behavior may be determined at least in partthrough machine learning associated with previous user behavior. Thebehavior may be determined to be a user contact with an object, whereinthe second light field data may be located to be concurrent with alocation of the object. The object may be a physical object in asurrounding environment. The object may be a virtual object insurrounding virtual environment. The user device may be communicativelycoupled to a display for presentation of display data. The sensor datamay include position sensor data. The sensor data may include motionsensor data.

In an aspect, a method may include receiving, by a computing deviceassociated with an image processor, a sensor data associated with a userdevice; generating, by the computing device, light field data based on apredicted behavior determined at least in part by the received sensordata; and communicating, by the computing device, the light field datato the user device. In embodiments, the light field data may include athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points. The predictedbehavior may be a predicted future position of the user device. Thesensor data may represent the predicted future position of the userdevice based on a velocity vector or acceleration vector determined fromthe sensor data. The predicted behavior may be a predicted viewing angleof the user device. The predicted viewing angle may be based on a lineof sight vector determined from the sensor data. The predicted behaviormay be based on a prior action. The prior action may be determined tohave similar behavior characteristics to a present behavior. Thepredicted behavior may be further based on an action of a second userdevice. The second user device may be determined to be in a surroundingenvironment through the sensor data. The second user device may beassociated with a virtual user. The light field data may include avolume of space proximate to the user device and the volume of space maybe determined by the predicted behavior. The light field data mayinclude a light field resolution and the light field resolution may bedetermined by the predicted behavior. The light field data may include adirectionality with respect to a user device perspective and thedirectionality may be determined by the predicted behavior. The lightfield data may include a light field point density and the light fieldpoint density may be determined by the predicted behavior. The userdevice may be a head-worn device, and the predicted behavior may beassociated with a head position of the user device. The predictedbehavior may be determined at least in part through machine learningassociated with previous behavior. The predicted behavior may bedetermined to be associated with a user contact with an object. Theobject may be a physical object in a surrounding environment. The objectmay be a virtual object in surrounding virtual environment. The userdevice may be communicatively coupled to a display for presentation ofthe display data. The sensor data may include position sensor data ormotion sensor data.

In an aspect, a system may include a computing device associated with animage processor, the computing device configured to store a set ofinstructions that, when executed, cause the computing device to: receivea sensor data associated with a user device; generate, by the computingdevice, light field data based on a predicted behavior determined atleast in part by the received sensor data; and communicate, by thecomputing device, the light field data to the user device. Inembodiments, the predicted behavior may be a behavior of the userdevice. The predicted behavior may be a predicted future position. Thepredicted behavior may be a predicted viewing angle. The predictedbehavior may be based on a prior action. The light field data mayinclude a three-dimensional volume describing the light flowing in aplurality of directions through a plurality of light field data points.The predicted behavior may be a predicted future position of the userdevice. The sensor data may represent the predicted future position ofthe user device based on a velocity vector or acceleration vectordetermined from the sensor data. The predicted behavior may be apredicted viewing angle of the user device. The predicted viewing anglemay be based on a line of sight vector determined from the sensor data.The predicted behavior may be based on a prior action. The prior actionmay be determined to have similar behavior characteristics to a presentbehavior. The predicted behavior may be further based on an action of asecond user device. The second user device may be determined to be in asurrounding environment through the sensor data. The second user devicemay be associated with a virtual user. The light field data may includea volume of space proximate to the user device and the volume of spacemay be determined by the predicted behavior. The light field data mayinclude a light field resolution and the light field resolution may bedetermined by the predicted behavior. The light field data may include adirectionality with respect to a user device perspective and thedirectionality may be determined by the predicted behavior. The lightfield data may include a light field point density and the light fieldpoint density may be determined by the predicted behavior. The userdevice may be a head-worn device, and the predicted behavior may beassociated with a head position of the user device. The predictedbehavior may be determined at least in part through machine learningassociated with previous behavior. The predicted behavior may bedetermined to be associated with a user contact with an object. Theobject may be a physical object in a surrounding environment. The objectmay be a virtual object in surrounding virtual environment. The userdevice may be communicatively coupled to a display for presentation ofthe display data. The sensor data may include position sensor data ormotion sensor data.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 depicts an embodiment of an e-commerce platform.

FIG. 2 depicts an embodiment for a home page of an administrator.

FIG. 3 depicts a prior art visual-based interactive system.

FIG. 4 depicts an embodiment visual-based interactive system.

FIG. 5 depicts an embodiment visual-based interactive system using lightfield data.

FIGS. 6A-6B depict a side and top view of ray tracing with respect tolight field data.

FIGS. 7A-7B depict a side and top view of ray tracking with respect tolight field data.

FIG. 8 depicts an embodiment visual-based interactive system using lightfield data in a partial view of surroundings.

FIG. 9 depicts an embodiment visual-based interactive system using lightfield data in a partial view of surroundings where some partial viewprocessing is performed on the user device.

FIG. 10 depicts an embodiment visual-based interactive system usinglight field data and predicted behavior.

DETAILED DESCRIPTION

The present disclosure will now be described in detail by describingvarious illustrative, non-limiting embodiments thereof with reference tothe accompanying drawings and exhibits. The disclosure may, however, beembodied in many different forms and should not be construed as beinglimited to the illustrative embodiments set forth herein. Rather, theembodiments are provided so that this disclosure will be thorough andwill fully convey the concept of the disclosure to those skilled in theart.

With reference to FIG. 1, an embodiment e-commerce platform 100 isdepicted for providing merchant products and services to customers.While the disclosure throughout contemplates using the apparatus,system, and process disclosed to purchase products and services, forsimplicity the description herein will refer to products. All referencesto products throughout this disclosure should also be understood to bereferences to products and/or services, including physical products,digital content, tickets, subscriptions, services to be provided, andthe like.

While the disclosure throughout contemplates that a ‘merchant’ and a‘customer’ may be more than individuals, for simplicity the descriptionherein may generally refer to merchants and customers as such. Allreferences to merchants and customers throughout this disclosure shouldalso be understood to be references to groups of individuals, companies,corporations, computing entities, and the like, and may representfor-profit or not-for-profit exchange of products. Further, while thedisclosure throughout refers to ‘merchants’ and ‘customers’, anddescribes their roles as such, the e-commerce platform 100 should beunderstood to more generally support users in an e-commerce environment,and all references to merchants and customers throughout this disclosureshould also be understood to be references to users, such as where auser is a merchant-user (e.g., a seller, retailer, wholesaler, orprovider of products), a customer-user (e.g., a buyer, purchase agent,or user of products), a prospective user (e.g., a user browsing and notyet committed to a purchase, a user evaluating the e-commerce platform100 for potential use in marketing and selling products, and the like),a service provider user (e.g., a shipping provider 112, a financialprovider, and the like), a company or corporate user (e.g., a companyrepresentative for purchase, sales, or use of products; an enterpriseuser; a customer relations or customer management agent, and the like),an information technology user, a computing entity user (e.g., acomputing bot for purchase, sales, or use of products), and the like.

The e-commerce platform 100 may provide a centralized system forproviding merchants with online resources and facilities for managingtheir business. The facilities described herein may be deployed in partor in whole through a machine that executes computer software, modules,program codes, and/or instructions on one or more processors which maybe part of or external to the platform 100. Merchants may utilize thee-commerce platform 100 for managing commerce with customers, such as byimplementing an e-commerce experience with customers through an onlinestore 138, through channels 110A-B, through POS devices 152 in physicallocations (e.g., a physical storefront or other location such as througha kiosk, terminal, reader, printer, 3D printer, and the like), bymanaging their business through the e-commerce platform 100, and byinteracting with customers through a communications facility 129 of thee-commerce platform 100, or any combination thereof. A merchant mayutilize the e-commerce platform 100 as a sole commerce presence withcustomers, or in conjunction with other merchant commerce facilities,such as through a physical store (e.g., ‘brick-and-mortar’ retailstores), a merchant off-platform website 104 (e.g., a commerce Internetwebsite or other internet or web property or asset supported by or onbehalf of the merchant separately from the e-commerce platform), and thelike. However, even these ‘other’ merchant commerce facilities may beincorporated into the e-commerce platform, such as where POS devices 152in a physical store of a merchant are linked into the e-commerceplatform 100, where a merchant off-platform website 104 is tied into thee-commerce platform 100, such as through ‘buy buttons’ that link contentfrom the merchant off platform website 104 to the online store 138, andthe like.

The online store 138 may represent a multitenant facility comprising aplurality of virtual storefronts. In embodiments, merchants may manageone or more storefronts in the online store 138, such as through amerchant device 102 (e.g., computer, laptop computer, mobile computingdevice, and the like), and offer products to customers through a numberof different channels 110A-B (e.g., an online store 138; a physicalstorefront through a POS device 152; electronic marketplace, through anelectronic buy button integrated into a website or social media channelsuch as on a social network, social media page, social media messagingsystem; and the like). A merchant may sell across channels 110A-B andthen manage their sales through the e-commerce platform 100, wherechannels 110A may be provided internal to the e-commerce platform 100 orfrom outside the e-commerce channel 110B. A merchant may sell in theirphysical retail store, at pop ups, through wholesale, over the phone,and the like, and then manage their sales through the e-commerceplatform 100. A merchant may employ all or any combination of these,such as maintaining a business through a physical storefront utilizingPOS devices 152, maintaining a virtual storefront through the onlinestore 138, and utilizing a communication facility 129 to leveragecustomer interactions and analytics 132 to improve the probability ofsales. Throughout this disclosure the terms online store 138 andstorefront may be used synonymously to refer to a merchant's onlinee-commerce offering presence through the e-commerce platform 100, wherean online store 138 may refer to the multitenant collection ofstorefronts supported by the e-commerce platform 100 (e.g., for aplurality of merchants) or to an individual merchant's storefront (e.g.,a merchant's online store).

In embodiments, a customer may interact through a customer device 150(e.g., computer, laptop computer, mobile computing device, and thelike), a POS device 152 (e.g., retail device, a kiosk, an automatedcheckout system, and the like), or any other commerce interface deviceknown in the art. The e-commerce platform 100 may enable merchants toreach customers through the online store 138, through POS devices 152 inphysical locations (e.g., a merchant's storefront or elsewhere), topromote commerce with customers through dialog via electroniccommunication facility 129, and the like, providing a system forreaching customers and facilitating merchant services for the real orvirtual pathways available for reaching and interacting with customers.

In embodiments, and as described further herein, the e-commerce platform100 may be implemented through a processing facility including aprocessor and a memory, the processing facility storing a set ofinstructions that, when executed, cause the e-commerce platform 100 toperform the e-commerce and support functions as described herein. Theprocessing facility may be part of a server, client, networkinfrastructure, mobile computing platform, cloud computing platform,stationary computing platform, or other computing platform, and provideelectronic connectivity and communications between and amongst theelectronic components of the e-commerce platform 100, merchant devices102, payment gateways 106, application developers, channels 110A-B,shipping providers 112, customer devices 150, point of sale devices 152,and the like. The e-commerce platform 100 may be implemented as a cloudcomputing service, a software as a service (SaaS), infrastructure as aservice (IaaS), platform as a service (PaaS), desktop as a Service(DaaS), managed software as a service (MSaaS), mobile backend as aservice (MBaaS), information technology management as a service(ITMaaS), and the like, such as in a software and delivery model inwhich software is licensed on a subscription basis and centrally hosted(e.g., accessed by users using a client (for example, a thin client) viaa web browser or other application, accessed through by POS devices, andthe like). In embodiments, elements of the e-commerce platform 100 maybe implemented to operate on various platforms and operating systems,such as iOS, Android, on the web, and the like (e.g., the administrator114 being implemented in multiple instances for a given online store foriOS, Android, and for the web, each with similar functionality).

In embodiments, the online store 138 may be served to a customer device150 through a webpage provided by a server of the e-commerce platform100. The server may receive a request for the webpage from a browser orother application installed on the customer device 150, where thebrowser (or other application) connects to the server through an IPAddress, the IP address obtained by translating a domain name. Inreturn, the server sends back the requested webpage. Webpages may bewritten in or include Hypertext Markup Language (HTML), templatelanguage, JavaScript, and the like, or any combination thereof. Forinstance, HTML is a computer language that describes static informationfor the webpage, such as the layout, format, and content of the webpage.Website designers and developers may use the template language to buildwebpages that combine static content, which is the same on multiplepages, and dynamic content, which changes from one page to the next. Atemplate language may make it possible to re-use the static elementsthat define the layout of a webpage, while dynamically populating thepage with data from an online store. The static elements may be writtenin HTML, and the dynamic elements written in the template language. Thetemplate language elements in a file may act as placeholders, such thatthe code in the file is compiled and sent to the customer device 150 andthen the template language is replaced by data from the online store138, such as when a theme is installed. The template and themes mayconsider tags, objects, and filters. The client device web browser (orother application) then renders the page accordingly.

In embodiments, online stores 138 may be served by the e-commerceplatform 100 to customers, where customers can browse and purchase thevarious products available (e.g., add them to a cart, purchaseimmediately through a buy-button, and the like). Online stores 138 maybe served to customers in a transparent fashion without customersnecessarily being aware that it is being provided through the e-commerceplatform 100 (rather than directly from the merchant). Merchants may usea merchant configurable domain name, a customizable HTML theme, and thelike, to customize their online store 138. Merchants may customize thelook and feel of their website through a theme system, such as wheremerchants can select and change the look and feel of their online store138 by changing their theme while having the same underlying product andbusiness data shown within the online store's product hierarchy. Themesmay be further customized through a theme editor, a design interfacethat enables users to customize their website's design with flexibility.Themes may also be customized using theme-specific settings that changeaspects, such as specific colors, fonts, and pre-built layout schemes.The online store may implement a content management system for websitecontent. Merchants may author blog posts or static pages and publishthem to their online store 138, such as through blogs, articles, and thelike, as well as configure navigation menus. Merchants may upload images(e.g., for products), video, content, data, and the like to thee-commerce platform 100, such as for storage by the system (e.g. as data134). In embodiments, the e-commerce platform 100 may provide functionsfor resizing images, associating an image with a product, adding andassociating text with an image, adding an image for a new productvariant, protecting images, and the like.

As described herein, the e-commerce platform 100 may provide merchantswith transactional facilities for products through a number of differentchannels 110A-B, including the online store 138, over the telephone, aswell as through physical POS devices 152 as described herein. Thee-commerce platform 100 may include business support services 116, anadministrator 114, and the like associated with running an on-linebusiness, such as providing a domain service 118 associated with theironline store, payment services 120 for facilitating transactions with acustomer, shipping services 122 for providing customer shipping optionsfor purchased products, risk and insurance services 124 associated withproduct protection and liability, merchant billing, and the like.Services 116 may be provided via the e-commerce platform 100 or inassociation with external facilities, such as through a payment gateway106 for payment processing, shipping providers 112 for expediting theshipment of products, and the like.

In embodiments, the e-commerce platform 100 may provide for integratedshipping services 122 (e.g., through an e-commerce platform shippingfacility or through a third-party shipping carrier), such as providingmerchants with real-time updates, tracking, automatic rate calculation,bulk order preparation, label printing, and the like.

FIG. 2 depicts a non-limiting embodiment for a home page of anadministrator 114, which may show information about daily tasks, astore's recent activity, and the next steps a merchant can take to buildtheir business. In embodiments, a merchant may log in to administrator114 via a merchant device 102 such as from a desktop computer or mobiledevice, and manage aspects of their online store 138, such as viewingthe online store's 138 recent activity, updating the online store's 138catalog, managing orders, recent visits activity, total orders activity,and the like. In embodiments, the merchant may be able to access thedifferent sections of administrator 114 by using the sidebar, such asshown on FIG. 2. Sections of the administrator 114 may include variousinterfaces for accessing and managing core aspects of a merchant'sbusiness, including orders, products, customers, available reports anddiscounts. The administrator 114 may also include interfaces formanaging sales channels for a store including the online store, mobileapplication(s) made available to customers for accessing the store(Mobile App), POS devices, and/or a buy button. The administrator 114may also include interfaces for managing applications (Apps) installedon the merchant's account; settings applied to a merchant's online store138 and account. A merchant may use a search bar to find products,pages, or other information. Depending on the device 102 or softwareapplication the merchant is using, they may be enabled for differentfunctionality through the administrator 114. For instance, if a merchantlogs in to the administrator 114 from a browser, they may be able tomanage all aspects of their online store 138. If the merchant logs infrom their mobile device (e.g. via a mobile application), they may beable to view all or a subset of the aspects of their online store 138,such as viewing the online store's 138 recent activity, updating theonline store's 138 catalog, managing orders, and the like.

More detailed information about commerce and visitors to a merchant'sonline store 138 may be viewed through acquisition reports or metrics,such as displaying a sales summary for the merchant's overall business,specific sales and engagement data for active sales channels, and thelike. Reports may include, acquisition reports, behavior reports,customer reports, finance reports, marketing reports, sales reports,custom reports, and the like. The merchant may be able to view salesdata for different channels 110A-B from different periods of time (e.g.,days, weeks, months, and the like), such as by using drop-down menus. Anoverview dashboard may be provided for a merchant that wants a moredetailed view of the store's sales and engagement data. An activity feedin the home metrics section may be provided to illustrate an overview ofthe activity on the merchant's account. For example, by clicking on a‘view all recent activity’ dashboard button, the merchant may be able tosee a longer feed of recent activity on their account. A home page mayshow notifications about the merchant's online store 138, such as basedon account status, growth, recent customer activity, and the like.Notifications may be provided to assist a merchant with navigatingthrough a process, such as capturing a payment, marking an order asfulfilled, archiving an order that is complete, and the like.

The e-commerce platform 100 may provide for a communications facility129 and associated merchant interface for providing electroniccommunications and marketing, such as utilizing an electronic messagingaggregation facility for collecting and analyzing communicationinteractions between merchants, customers, merchant devices 102,customer devices 150, POS devices 152, and the like, to aggregate andanalyze the communications, such as for increasing the potential forproviding a sale of a product, and the like. For instance, a customermay have a question related to a product, which may produce a dialogbetween the customer and the merchant (or automated processor-basedagent representing the merchant), where the communications facility 129analyzes the interaction and provides analysis to the merchant on how toimprove the probability for a sale.

The e-commerce platform 100 may provide a financial facility 120 forsecure financial transactions with customers, such as through a securecard server environment. The e-commerce platform 100 may store creditcard information, such as in payment card industry data (PCI)environments (e.g., a card server), to reconcile financials, billmerchants, perform automated clearing house (ACH) transfers between ane-commerce platform 100 financial institution account and a merchant'sback account (e.g., when using capital), and the like. These systems mayhave Sarbanes-Oxley Act (SOX) compliance and a high level of diligencerequired in their development and operation. The financial facility 120may also provide merchants with financial support, such as through thelending of capital (e.g., lending funds, cash advances, and the like)and provision of insurance. In addition, the e-commerce platform 100 mayprovide for a set of marketing and partner services and control therelationship between the e-commerce platform 100 and partners. They alsomay connect and onboard new merchants with the e-commerce platform 100.These services may enable merchant growth by making it easier formerchants to work across the e-commerce platform 100. Through theseservices, merchants may be provided help facilities via the e-commerceplatform 100.

In embodiments, online store 138 may support a great number ofindependently administered storefronts and process a large volume oftransactional data on a daily basis for a variety of products.Transactional data may include customer contact information, billinginformation, shipping information, information on products purchased,information on services rendered, and any other information associatedwith business through the e-commerce platform 100. In embodiments, thee-commerce platform 100 may store this data in a data facility 134. Thetransactional data may be processed to produce analytics 132, which inturn may be provided to merchants or third-party commerce entities, suchas providing consumer trends, marketing and sales insights,recommendations for improving sales, evaluation of customer behaviors,marketing and sales modeling, trends in fraud, and the like, related toonline commerce, and provided through dashboard interfaces, throughreports, and the like. The e-commerce platform 100 may store informationabout business and merchant transactions, and the data facility 134 mayhave many ways of enhancing, contributing, refining, and extractingdata, where over time the collected data may enable improvements toaspects of the e-commerce platform 100.

Referring again to FIG. 1, in embodiments the e-commerce platform 100may be configured with a commerce management engine 136 for contentmanagement, task automation and data management to enable support andservices to the plurality of online stores 138 (e.g., related toproducts, inventory, customers, orders, collaboration, suppliers,reports, financials, risk and fraud, and the like), but be extensiblethrough applications 142A-B that enable greater flexibility and customprocesses required for accommodating an ever-growing variety of merchantonline stores, POS devices, products, and services, where applications142A may be provided internal to the e-commerce platform 100 orapplications 142B from outside the e-commerce platform 100. Inembodiments, an application 142A may be provided by the same partyproviding the platform 100 or by a different party. In embodiments, anapplication 142B may be provided by the same party providing theplatform 100 or by a different party. The commerce management engine 136may be configured for flexibility and scalability through portioning(e.g., sharding) of functions and data, such as by customer identifier,order identifier, online store identifier, and the like. The commercemanagement engine 136 may accommodate store-specific business logic andin some embodiments, may incorporate the administrator 114 and/or theonline store 138.

The commerce management engine 136 includes base or “core” functions ofthe e-commerce platform 100, and as such, as described herein, not allfunctions supporting online stores 138 may be appropriate for inclusion.For instance, functions for inclusion into the commerce managementengine 136 may need to exceed a core functionality threshold throughwhich it may be determined that the function is core to a commerceexperience (e.g., common to a majority of online store activity, such asacross channels, administrator interfaces, merchant locations,industries, product types, and the like), is re-usable across onlinestores 138 (e.g., functions that can be re-used/modified across corefunctions), limited to the context of a single online store 138 at atime (e.g., implementing an online store ‘isolation principle’, wherecode should not be able to interact with multiple online stores 138 at atime, ensuring that online stores 138 cannot access each other's data),provide a transactional workload, and the like. Maintaining control ofwhat functions are implemented may enable the commerce management engine136 to remain responsive, as many required features are either serveddirectly by the commerce management engine 136 or enabled through aninterface 140A-B, such as by its extension through an applicationprogramming interface (API) connection to applications 142A-B andchannels 110A-B, where interfaces 140A may be provided to applications142A and/or channels 110A inside the e-commerce platform 100 or throughinterfaces 140B provided to applications 142B and/or channels 110Boutside the e-commerce platform 100. Generally, the platform 100 mayinclude interfaces 140A-B (which may be extensions, connectors, APIs,and the like) which facilitate connections to and communications withother platforms, systems, software, data sources, code and the like.Such interfaces 140A-B may be an interface 140A of the commercemanagement engine 136 or an interface 140B of the platform 100 moregenerally. If care is not given to restricting functionality in thecommerce management engine 136, responsiveness could be compromised,such as through infrastructure degradation through slow databases ornon-critical backend failures, through catastrophic infrastructurefailure such as with a data center going offline, through new code beingdeployed that takes longer to execute than expected, and the like. Toprevent or mitigate these situations, the commerce management engine 136may be configured to maintain responsiveness, such as throughconfiguration that utilizes timeouts, queues, back-pressure to preventdegradation, and the like.

Although isolating online store data is important to maintaining dataprivacy between online stores 138 and merchants, there may be reasonsfor collecting and using cross-store data, such as for example, with anorder risk assessment system or a platform payment facility, both ofwhich require information from multiple online stores 138 to performwell. In embodiments, rather than violating the isolation principle, itmay be preferred to move these components out of the commerce managementengine 136 and into their own infrastructure within the e-commerceplatform 100.

In embodiments, the e-commerce platform 100 may provide for a platformpayment facility 120, which is another example of a component thatutilizes data from the commerce management engine 136 but may be locatedoutside so as to not violate the isolation principle. The platformpayment facility 120 may allow customers interacting with online stores138 to have their payment information stored safely by the commercemanagement engine 136 such that they only have to enter it once. When acustomer visits a different online store 138, even if they've never beenthere before, the platform payment facility 120 may recall theirinformation to enable a more rapid and correct check out. This mayprovide a cross-platform network effect, where the e-commerce platform100 becomes more useful to its merchants as more merchants join, such asbecause there are more customers who checkout more often because of theease of use with respect to customer purchases. To maximize the effectof this network, payment information for a given customer may beretrievable from an online store's checkout, allowing information to bemade available globally across online stores 138. It would be difficultand error prone for each online store 138 to be able to connect to anyother online store 138 to retrieve the payment information stored there.As a result, the platform payment facility may be implemented externalto the commerce management engine 136.

For those functions that are not included within the commerce managementengine 136, applications 142A-B provide a way to add features to thee-commerce platform 100. Applications 142A-B may be able to access andmodify data on a merchant's online store 138, perform tasks through theadministrator 114, create new flows for a merchant through a userinterface (e.g., that is surfaced through extensions/API), and the like.Merchants may be enabled to discover and install applications 142A-Bthrough application search, recommendations, and support 128. Inembodiments, core products, core extension points, applications, and theadministrator 114 may be developed to work together. For instance,application extension points may be built inside the administrator 114so that core features may be extended by way of applications, which maydeliver functionality to a merchant through the extension.

In embodiments, applications 142A-B may deliver functionality to amerchant through the interface 140A-B, such as where an application142A-B is able to surface transaction data to a merchant (e.g., App:“Engine, surface my app data in mobile and web admin using the embeddedapp SDK”), and/or where the commerce management engine 136 is able toask the application to perform work on demand (Engine: “App, give me alocal tax calculation for this checkout”).

Applications 142A-B may support online stores 138 and channels 110A-B,provide for merchant support, integrate with other services, and thelike. Where the commerce management engine 136 may provide thefoundation of services to the online store 138, the applications 142A-Bmay provide a way for merchants to satisfy specific and sometimes uniqueneeds. Different merchants will have different needs, and so may benefitfrom different applications 142A-B. Applications 142A-B may be betterdiscovered through the e-commerce platform 100 through development of anapplication taxonomy (categories) that enable applications to be taggedaccording to a type of function it performs for a merchant; throughapplication data services that support searching, ranking, andrecommendation models; through application discovery interfaces such asan application store, home information cards, an application settingspage; and the like.

Applications 142A-B may be connected to the commerce management engine136 through an interface 140A-B, such as utilizing APIs to expose thefunctionality and data available through and within the commercemanagement engine 136 to the functionality of applications (e.g.,through REST, GraphQL, and the like). For instance, the e-commerceplatform 100 may provide API interfaces 140A-B to merchant andpartner-facing products and services, such as including applicationextensions, process flow services, developer-facing resources, and thelike. With customers more frequently using mobile devices for shopping,applications 142A-B related to mobile use may benefit from moreextensive use of APIs to support the related growing commerce traffic.The flexibility offered through use of applications and APIs (e.g., asoffered for application development) enable the e-commerce platform 100to better accommodate new and unique needs of merchants (and internaldevelopers through internal APIs) without requiring constant change tothe commerce management engine 136, thus providing merchants what theyneed when they need it. For instance, shipping services 122 may beintegrated with the commerce management engine 136 through a shipping orcarrier service API, thus enabling the e-commerce platform 100 toprovide shipping service functionality without directly impacting coderunning in the commerce management engine 136.

Many merchant problems may be solved by letting partners improve andextend merchant workflows through application development, such asproblems associated with back-office operations (merchant-facingapplications 142A-B) and in the online store 138 (customer-facingapplications 142A-B). As a part of doing business, many merchants willuse mobile and web related applications on a daily basis for back-officetasks (e.g., merchandising, inventory, discounts, fulfillment, and thelike) and online store tasks (e.g., applications related to their onlineshop, for flash-sales, new product offerings, and the like), whereapplications 142A-B, through extension/API 140A-B, help make productseasy to view and purchase in a fast growing marketplace. In embodiments,partners, application developers, internal applications facilities, andthe like, may be provided with a software development kit (SDK), such asthrough creating a frame within the administrator 114 that sandboxes anapplication interface. In embodiments, the administrator 114 may nothave control over nor be aware of what happens within the frame. The SDKmay be used in conjunction with a user interface kit to produceinterfaces that mimic the look and feel of the e-commerce platform 100,such as acting as an extension of the commerce management engine 136.

Applications 142A-B that utilize APIs may pull data on demand, but oftenthey also need to have data pushed when updates occur. Update events maybe implemented in a subscription model, such as for example, customercreation, product changes, or order cancelation. Update events mayprovide merchants with needed updates with respect to a changed state ofthe commerce management engine 136, such as for synchronizing a localdatabase, notifying an external integration partner, and the like.Update events may enable this functionality without having to poll thecommerce management engine 136 all the time to check for updates, suchas through an update event subscription. In embodiments, when a changerelated to an update event subscription occurs, the commerce managementengine 136 may post a request, such as to a predefined callback URL. Thebody of this request may contain a new state of the object and adescription of the action or event. Update event subscriptions may becreated manually, in the administrator facility 114, or automatically(e.g., via the API 140A-B). In embodiments, update events may be queuedand processed asynchronously from a state change that triggered them,which may produce an update event notification that is not distributedin real-time.

In embodiments, the e-commerce platform 100 may provide applicationsearch, recommendation and support 128. Application search,recommendation and support 128 may include developer products and toolsto aid in the development of applications, an application dashboard(e.g., to provide developers with a development interface, toadministrators for management of applications, to merchants forcustomization of applications, and the like), facilities for installingand providing permissions with respect to providing access to anapplication 142A-B (e.g., for public access, such as where criteria mustbe met before being installed, or for private use by a merchant),application searching to make it easy for a merchant to search forapplications 142A-B that satisfy a need for their online store 138,application recommendations to provide merchants with suggestions on howthey can improve the user experience through their online store 138, adescription of core application capabilities within the commercemanagement engine 136, and the like. These support facilities may beutilized by application development performed by any entity, includingthe merchant developing their own application 142A-B, a third-partydeveloper developing an application 142A-B (e.g., contracted by amerchant, developed on their own to offer to the public, contracted foruse in association with the e-commerce platform 100, and the like), oran application 142A or 142B being developed by internal personalresources associated with the e-commerce platform 100. In embodiments,applications 142A-B may be assigned an application identifier (ID), suchas for linking to an application (e.g., through an API), searching foran application, making application recommendations, and the like.

The commerce management engine 136 may include base functions of thee-commerce platform 100 and expose these functions through APIs 140A-Bto applications 142A-B. The APIs 140A-B may enable different types ofapplications built through application development. Applications 142A-Bmay be capable of satisfying a great variety of needs for merchants butmay be grouped roughly into three categories: customer-facingapplications, merchant-facing applications, integration applications,and the like. Customer-facing applications 142A-B may include onlinestore 138 or channels 110A-B that are places where merchants can listproducts and have them purchased (e.g., the online store, applicationsfor flash sales (e.g., merchant products or from opportunistic salesopportunities from third-party sources), a mobile store application, asocial media channel, an application for providing wholesale purchasing,and the like). Merchant-facing applications 142A-B may includeapplications that allow the merchant to administer their online store138 (e.g., through applications related to the web or website or tomobile devices), run their business (e.g., through applications relatedto POS devices), to grow their business (e.g., through applicationsrelated to shipping (e.g., drop shipping), use of automated agents, useof process flow development and improvements), and the like. Integrationapplications may include applications that provide useful integrationsthat participate in the running of a business, such as shippingproviders 112 and payment gateways.

In embodiments, an application developer may use an application proxy tofetch data from an outside location and display it on the page of anonline store 138. Content on these proxy pages may be dynamic, capableof being updated, and the like. Application proxies may be useful fordisplaying image galleries, statistics, custom forms, and other kinds ofdynamic content. The core-application structure of the e-commerceplatform 100 may allow for an increasing number of merchant experiencesto be built in applications 142A-B so that the commerce managementengine 136 can remain focused on the more commonly utilized businesslogic of commerce.

The e-commerce platform 100 provides an online shopping experiencethrough a curated system architecture that enables merchants to connectwith customers in a flexible and transparent manner. A typical customerexperience may be better understood through an embodiment examplepurchase workflow, where the customer browses the merchant's products ona channel 110A-B, adds what they intend to buy to their cart, proceedsto checkout, and pays for the content of their cart resulting in thecreation of an order for the merchant. The merchant may then review andfulfill (or cancel) the order. The product is then delivered to thecustomer. If the customer is not satisfied, they might return theproducts to the merchant.

In an example embodiment, a customer may browse a merchant's products ona channel 110A-B. A channel 110A-B is a place where customers can viewand buy products. In embodiments, channels 110A-B may be modeled asapplications 142A-B (a possible exception being the online store 138,which is integrated within the commence management engine 136). Amerchandising component may allow merchants to describe what they wantto sell and where they sell it. The association between a product and achannel may be modeled as a product publication and accessed by channelapplications, such as via a product listing API. A product may have manyoptions, like size and color, and many variants that expand theavailable options into specific combinations of all the options, likethe variant that is extra-small and green, or the variant that is sizelarge and blue. Products may have at least one variant (e.g., a “defaultvariant” is created for a product without any options). To facilitatebrowsing and management, products may be grouped into collections,provided product identifiers (e.g., stock keeping unit (SKU)) and thelike. Collections of products may be built by either manuallycategorizing products into one (e.g., a custom collection), by buildingrulesets for automatic classification (e.g., a smart collection), andthe like. Products may be viewed as 2D images, 3D images, rotating viewimages, through a virtual or augmented reality interface, and the like.

In embodiments, the customer may add what they intend to buy to theircart (in an alternate embodiment, a product may be purchased directly,such as through a buy button as described herein). Customers may addproduct variants to their shopping cart. The shopping cart model may bechannel specific. The online store 138 cart may be composed of multiplecart line items, where each cart line item tracks the quantity for aproduct variant. Merchants may use cart scripts to offer specialpromotions to customers based on the content of their cart. Since addinga product to a cart does not imply any commitment from the customer orthe merchant, and the expected lifespan of a cart may be in the order ofminutes (not days), carts may be persisted to an ephemeral data store.

The customer then proceeds to checkout. A checkout component mayimplement a web checkout as a customer-facing order creation process. Acheckout API may be provided as a computer-facing order creation processused by some channel applications to create orders on behalf ofcustomers (e.g., for point of sale). Checkouts may be created from acart and record a customer's information such as email address, billing,and shipping details. On checkout, the merchant commits to pricing. Ifthe customer inputs their contact information but does not proceed topayment, the e-commerce platform 100 may provide an opportunity tore-engage the customer (e.g., in an abandoned checkout feature). Forthose reasons, checkouts can have much longer lifespans than carts(hours or even days) and are therefore persisted. Checkouts maycalculate taxes and shipping costs based on the customer's shippingaddress. Checkout may delegate the calculation of taxes to a taxcomponent and the calculation of shipping costs to a delivery component.A pricing component may enable merchants to create discount codes (e.g.,‘secret’ strings that when entered on the checkout apply new prices tothe items in the checkout). Discounts may be used by merchants toattract customers and assess the performance of marketing campaigns.Discounts and other custom price systems may be implemented on top ofthe same platform piece, such as through price rules (e.g., a set ofprerequisites that when met imply a set of entitlements). For instance,prerequisites may be items such as “the order subtotal is greater than$100” or “the shipping cost is under $10”, and entitlements may be itemssuch as “a 20% discount on the whole order” or “$10 off products X, Y,and Z”.

Customers then pay for the content of their cart resulting in thecreation of an order for the merchant. Channels 110A-B may use thecommerce management engine 136 to move money, currency or a store ofvalue (such as dollars or a cryptocurrency) to and from customers andmerchants. Communication with the various payment providers (e.g.,online payment systems, mobile payment systems, digital wallet, creditcard gateways, and the like) may be implemented within a paymentprocessing component. The actual interactions with the payment gateways106 may be provided through a card server environment. In embodiments,the payment gateway 106 may accept international payment, such asintegrating with leading international credit card processors. The cardserver environment may include a card server application, card sink,hosted fields, and the like. This environment may act as the securegatekeeper of the sensitive credit card information. In embodiments,most of the process may be orchestrated by a payment processing job. Thecommerce management engine 136 may support many other payment methods,such as through an offsite payment gateway 106 (e.g., where the customeris redirected to another website), manually (e.g., cash), online paymentmethods (e.g., online payment systems, mobile payment systems, digitalwallet, credit card gateways, and the like), gift cards, and the like.At the end of the checkout process, an order is created. An order is acontract of sale between the merchant and the customer where themerchant agrees to provide the goods and services listed on the orders(e.g., order line items, shipping line items, and the like) and thecustomer agrees to provide payment (including taxes). This process maybe modeled in a sales component. Channels 110A-B that do not rely oncommerce management engine 136 checkouts may use an order API to createorders. Once an order is created, an order confirmation notification maybe sent to the customer and an order placed notification sent to themerchant via a notification component. Inventory may be reserved when apayment processing job starts to avoid over-selling (e.g., merchants maycontrol this behavior from the inventory policy of each variant).Inventory reservation may have a short time span (minutes) and may needto be very fast and scalable to support flash sales (e.g., a discount orpromotion offered for a short time, such as targeting impulse buying).The reservation is released if the payment fails. When the paymentsucceeds, and an order is created, the reservation is converted into along-term inventory commitment allocated to a specific location. Aninventory component may record where variants are stocked, and tracksquantities for variants that have inventory tracking enabled. It maydecouple product variants (a customer facing concept representing thetemplate of a product listing) from inventory items (a merchant facingconcept that represent an item whose quantity and location is managed).An inventory level component may keep track of quantities that areavailable for sale, committed to an order or incoming from an inventorytransfer component (e.g., from a vendor).

The merchant may then review and fulfill (or cancel) the order. A reviewcomponent may implement a business process merchant's use to ensureorders are suitable for fulfillment before actually fulfilling them.Orders may be fraudulent, require verification (e.g., ID checking), havea payment method which requires the merchant to wait to make sure theywill receive their funds, and the like. Risks and recommendations may bepersisted in an order risk model. Order risks may be generated from afraud detection tool, submitted by a third-party through an order riskAPI, and the like. Before proceeding to fulfillment, the merchant mayneed to capture the payment information (e.g., credit card information)or wait to receive it (e.g., via a bank transfer, check, and the like)and mark the order as paid. The merchant may now prepare the productsfor delivery. In embodiments, this business process may be implementedby a fulfillment component. The fulfillment component may group the lineitems of the order into a logical fulfillment unit of work based on aninventory location and fulfillment service. The merchant may review,adjust the unit of work, and trigger the relevant fulfillment services,such as through a manual fulfillment service (e.g., at merchant managedlocations) used when the merchant picks and packs the products in a box,purchase a shipping label and input its tracking number, or just markthe item as fulfilled. A custom fulfillment service may send an email(e.g., a location that doesn't provide an API connection). An APIfulfillment service may trigger a third party, where the third-partyapplication creates a fulfillment record. A legacy fulfillment servicemay trigger a custom API call from the commerce management engine 136 toa third party (e.g., fulfillment by Amazon). A gift card fulfillmentservice may provision (e.g., generating a number) and activate a giftcard. Merchants may use an order printer application to print packingslips. The fulfillment process may be executed when the items are packedin the box and ready for shipping, shipped, tracked, delivered, verifiedas received by the customer, and the like.

If the customer is not satisfied, they may be able to return theproduct(s) to the merchant. The business process merchants may gothrough to “un-sell” an item may be implemented by a return component.Returns may consist of a variety of different actions, such as arestock, where the product that was sold actually comes back into thebusiness and is sellable again; a refund, where the money that wascollected from the customer is partially or fully returned; anaccounting adjustment noting how much money was refunded (e.g.,including if there was any restocking fees, or goods that weren'treturned and remain in the customer's hands); and the like. A return mayrepresent a change to the contract of sale (e.g., the order), and wherethe e-commerce platform 100 may make the merchant aware of complianceissues with respect to legal obligations (e.g., with respect to taxes).In embodiments, the e-commerce platform 100 may enable merchants to keeptrack of changes to the contract of sales over time, such as implementedthrough a sales model component (e.g., an append-only date-based ledgerthat records sale-related events that happened to an item).

In embodiments, the e-commerce platform 100 may interface with a userdevice in support of interactive visual content being delivered to auser device, such as where the user device is a virtual reality (VR)device, augmented reality (AR) device, mixed reality (MR) device (e.g.,head-worn, body-worn, or hand-held), or any device that caninterpret/process visual content (e.g., a smart phone or mobilecomputing device), where the e-commerce platform 100 provides imageprocessing functions to generate the visual content to be delivered tothe user device in response to a change in visualization context. Inembodiments, the e-commerce platform 100 may be any platform orcomputing device remote to a user device. In embodiments, a user devicemay also have the ability to process video data and to detect userposition and direction of view (e.g., through sensors such as positionsensors, motion sensors, cameras, and the like). In embodiments, a userdevice may be a combination of smart phone, sensor array, and VR, AR, orMR devices, such as where a smart phone interfaces with a VR, AR, or MRdevice, or acts as a VR, AR, or MR device in conjunction with wornsensors or devices (e.g., a smart phone interfacing (e.g., viaBluetooth) with a head-worn sensor to detect position and/or motion of auser's head), and the like.

Although image processing functions associated with the processing ofvisual content may be provided by an image processor in association withthe e-commerce platform 100, one skilled in the art will appreciate thatany communicatively coupled hardware or software image processingcomputing facility may provide this function, such as provided through acloud platform, a network, a server, a computing device, an applicationproviding image processing capabilities, and the like, such as over theInternet using a WiFi or other wireless or wired connection, and thelike. For instance, image processing may be provided by the e-commerceplatform 100, remote from the e-commerce platform 100, or in anycombined or networked configuration known in the art.

With reference to FIG. 3, image processing 304 may be performed remotelyfrom a user device 306, and as such may result in a delay between theremote generation of video data and the displaying of the video data onthe user device 306 (e.g., due to image processing time, transmissiontime, and the like). Updating and/or sending video data may be generallyassociated with a scene change in the user experience delivered to theuser device 306, such as due to a virtual environmental change (e.g.,the leaves moving on a nearby tree due to the wind and the need to sendvideo data to the user device to update the scene for the movingleaves), another virtual user or object entering the scene (e.g., asecond user in the user's field of view), an environmental change due toan action taken by the user through the user device (e.g., the leaves onthe tree moving due to the user activating a virtual fan from the userdevice), a user device input received that senses a movement of the userdevice 306 (e.g., a head-mounted sensor that senses the user istranslating (e.g., the user moving forward causing the scene to change)or rotating (e.g., the user turning their head causing the field of viewof the scene to change), and the like. Although embodiments hereingenerally describe user inputs 302 as an interactive cause for the needto transmit new video data to the user device 306, one skilled in theart will appreciate that non-interactive causes (e.g., scene changesthat are not the direct result of a user action, such as dynamicmovements of the environment (e.g., the blowing of leaves on a tree))may also result in video data updates and/or transmission.

In embodiments a user input 302 (e.g., sensor data associated with theuser device) may be received by the e-commerce platform 100 for imageprocessing 304 (or other image processing) where visual content isgenerated and communicated to the user device 306. For instance, a usermay be using a VR user device where the user responds to visual content(or, in embodiments throughout, audio content where visualrepresentation has a corresponding auditory signal that may be processedindependently or concurrently) being displayed and where the userresponse is detected by a sensor (e.g., on the user device) andcommunicated back to the e-commerce platform 100 for image processing.The e-commerce platform 100 then generates a new visual content inresponse to the resulting scene change. In an example, the user may seea virtual door to the left as displayed by the VR device in a virtualenvironment where the user makes a motion due to the scene change. TheVR user device senses this movement (e.g., head motion or body motion)and sends related sensor data back to the e-commerce platform 100. Imageprocessing resources of the e-commerce platform 100 compute informationregarding the scene and video data is transmitted (e.g., over theinternet using a WiFi connection) to the VR device. The video data isthen further processed on the VR device to create a display for theuser.

In embodiments, the processing of the image content may be dividedbetween a remote image processing computing device (e.g., on a cloudplatform) and processing locally to the user device (e.g., on the userdevice itself). In embodiments, in order to minimize the resourcesrequired on the user device, much of the image processing functionalitymay be provided on the remote image processing computing device, withthe user device containing whatever processing is required to convertthe transmitted image content into display data. However, the time forthis process, even with minimal processing on the user device, mayresult in latency (e.g., from the time of the user input until the timethe resulting visual content is provided to the user device), such asfor a VR, AR, or MR user experience.

As a result, interactive visual content systems have explored thereduction of latency using services in wireless technologies such asenhanced mobile broadband (eMBB), ultra-reliable and low latencycommunication (URLLC), and the like, as well as the use of highlyspecialized network protocols to deliver visual content at a reducedlatency (e.g., <15 ms round trip). Techniques for combating virtualreality sickness include high-quality tracking systems to synchronizevirtual environment movements and the effect on the vestibular system.Another method to combat virtual reality sickness leverages themanipulation of field of view (FOV), either by reduction or changing theFOV in the user device display to change the perceived motion.Three-dimensional interactive elements may follow cloud renderingprinciples to leverage the large processing resources available bymoving more processing to the cloud as opposed to on the local userdevice. There may be non-zero latency in cloud rendering systems.Interactive headsets may handoff rendering processing to the cloud toonly display a video stream of the fully rendered scene. Video streamapproaches may leverage the efficiencies of H.264, H.265, AV1, and othercodecs to improve performance metrics. In addition, advances in videostreaming technologies may help to reduce transmission latency.

However, processed video streams are susceptible to network jitters andinterruptions (and general latency) which can lead to jitters andfreezes in the video stream leading to a degraded user experience (e.g.,conscious awareness of the delays and jitter). Additionally, any delayin rendering scene changes may result in virtual reality sickness (e.g.,often due to even unconscious delays, where there may be a subset of thepopulation that may notice latency no matter how small).

Referring to FIG. 4, to help solve the consequences of latencyassociated with transmitting video stream data, a computing device 402,such as including or in association with an image processor 408, maytransmit light field data 410 to the user device 412, such as inresponse to a scene change (e.g., change in the environment or due to auser input 404 such as sensor data 406). For instance, the light fielddata 410 may include a visual content that is greater than a displayfield of view of a user device with respect to position (e.g., thedisplay views a field of view for a current position but the light fielddata includes data for viewing from a position ahead, behind, or to theside of the current position), viewing angle (e.g., the display views a45-degree field of view but the light field data includes data forviewing out to 90 degrees, 180 degrees, or a full 360 degrees), and thelike, such as where the light field data 410 is for a greater field ofview at the time of the communication of the light field data, at a timeof the expected time of receipt of the light field data, and the like.For example, the light field data 410 may include data for displaying afield of view for a user in a current position and viewing angle butalso for a future or predicted position, if the user moves ahead, turnsaround, and the like. In embodiments, the light field data 410 mayinclude data for displaying a current field of view as well as a futurefield of view that is predicted with a certain probability (e.g., basedon current or past behavior it is likely the user will move forward andturn to the left). The light field data may include data for what a usercan see at a current position, but also for what the user may see if theuser takes a step forward (since what the user sees in the step forwardmay be outside the user's field of view from when the user is a stepback). As such, the user device will have light field data from thesurrounding environment to accommodate scene changes (e.g., translationsand rotations from a current position and viewing angle) without theneed to send additional data to the user device.

Although the present description may generally utilize an imageprocessor 408 in the processing of light field data 410, in practicalapplications communications between the image processor 408 and the userdevice 412 may be mediated by or through a computing device 402, such aswhere the computing device 402 is managing communications, processingdata associated with a visual-based interactive system using light fielddata 410 (e.g., utilizing an image processor 408, such as with or inassociation with the e-commerce platform 100), and/or a computing devicelocal to the user device (e.g., a local console, a smart phone, and thelike, communicating with the user device), for providing processingfunctions. As such, for simplicity the present description generallydepicts image processing and the communications with the user device 412to be through the computing device 402. However, this is not meant to belimiting in any way, where one skilled in the art will appreciate thatimage processing and/or communications with the user device 412 may beprovided by the computing device 402 remote from the user device orlocal to the user device, a local computing device, the image processor408 (e.g., provided in the cloud, the e-commerce platform, the localcomputing device), or in combination, where the computing device 402 maycomprise the image processor 408, communicate with the image processor408, and the like, in any computing configuration known in the art.

A light field may describe the amount of light flowing in everydirection through every point in space. In embodiments, the light fielddata 410 may be limited to a determined or configured volume of space oramount of light for a given volume such as the amount of light flowingthrough a given point in space may be limited to a number of rays oflight passing through the point, and/or the number of points per unit ofspace may be limited by a point density. However, by providing lightfield data 410 to a user device 412 the user device would have all thevisual information required for the user device to visualize the spaceas it is moved around within the volume of space without the need totransmit more visual content to the user device. That is, as long as theuser (operating the user device) stays within the set volume there maybe no need to transmit more data to the user device 412 in a relativelynon-changing visual scene. Thus, although the time to perform the imageprocessing steps at the computing device 402 and the time to transmitbetween the computing device 402 and the user device 412 may not havechanged, the perceived scene lag by a user may be reduced or eliminatedby controlling the volume of light field data 410 provided to the userdevice 412. If storage and processing on the user device 412 wasunlimited, the computing device 402 may only need to send one largelight field to the user device 412 per scene. Practically, the computingdevice 402 may provide a volume or series of volumes to the user device412 based on some criteria with respect to the current and/or predictedfuture user device position and movement, such as with respect to aviewing perspective of the user device 412, a velocity and/oracceleration vector of the user device 412, a behavior of the userdevice 412 (e.g., associated with the behavior of the user operating theuser device), nearby other user devices, and the like. Thus, using lightfield data 410 enables faster overall processing through light fieldrendering, such as based on sensors (e.g., accelerometers and the like)tuned to relatively small or micro movements of the user device 412(e.g., head movements of a VR or AR device) and may help reduceperceived latency. Transmitting light field data 410 may also reduce theinstances of virtual reality sickness by enabling a continuous virtualenvironment provided through the locally stored light field without thejarring environmental effects of network jitters or interruptionsassociated with transmission of traditional visual data (e.g., videostream data) for every user device movement.

In embodiments, light field data 410 (or series of light field data) ina stream of light fields may be described by any shaped light fieldsurface volume, such as cubes or cylinders of light field data 410, orpartial surface forms, where the light field data 410 points reside onlight field surface volume. That is, the ‘points’ of the light field areon this surface volume, but do not fill the surface volume. Thus, thelight field surface volume represents a potential display area for theuser device 412 for multiple views from within the light field surfacevolume (e.g., a user could turn their head around viewing in threedimensions from a full surface volume surrounding the user device, movewithin the surface volume, and the like). For example, the computingdevice 402 may send the user device 412 light field data 410 comprisinga light field volume surrounding the user device 412 in all directionsout to one meter. Alternatively, the computing device 402 mayselectively send light field data 410 for a portion of a full lightfield volume surrounding the user device 412, such as for a volumesurface or hemisphere of space surrounding the user device 412 centeredon the current direction of view presented by the user device 412 (e.g.,the user device is a VR headset and the light field data 410 is for ahemisphere centered on the field of view of the VR headset).

Referring to FIG. 5, a computing device 402 may utilize an imageprocessor 408 to process light field data 410 for communication to auser device 412. Although FIG. 5 depicts the image processor within thecomputing device 402, as described herein the image processor 408 may beincluded as part of the computing device 402, provided external to thecomputing device 402, or in combination. In an embodiment process flowexample, in a first step 510, which may be optional in embodiments, thecomputing device 402 may communicate a first light field data to theuser device 412. For instance, the first light field data may containviews for a first location of the user device 412 (e.g., where a user ofthe user device is standing, either in real or virtual space)concentrated in a first view (e.g., a display perspective of the userdevice representative of a direction a user of the user device islooking). In a second step 512 the computing device 402 may receivesensor data 406 associated with the user device 412. For instance, thesensor data 406 may be position or motion sensor data (e.g., from anaccelerometer mounted in such a way as to measure movements of the userdevice 412 or worn on the user to measure the movements of the user(e.g., head of the user)). In embodiments, sensor data 406 may be storedin a sensor data store 508, such as for future processing (e.g.,predicting a future sensor data from past sensor data). In a third step514 the computing device 402 may generate a second light field databased at least in part on the received sensor data 406. For instance,the sensor data 406 may have been processed and determined to representa change in viewing angle (e.g., the head of the user has turned) orposition of the user device 412 (e.g., the user device has movedforward). As a result, the computing device 402 may have generated thesecond light field data to include a light field volume for a differentview angle, a different position in space, and the like, based on thesensor data 406. In a fourth step 516, the computing device 402 maycommunicate the second light field data to the user device 412. Inembodiments, the user device 412 may include a processor for processingthe light field data 410, converting light field data 410 into displaydata for presentation on a display 520, collect sensor data 406associated with a sensor 522, and the like.

In embodiments, the light field data 410 may be restricted to a subsetor portion of a light field, such as in an AR or MR application, wherethe light field data 410 is restricted to a scene of interest (e.g., aportion of a full light field volume). In embodiments the determining ofthe subset or portion of the light field may be a pre-processing step(e.g., at the generation of the light field data), done at time oftransmission to determine what portion of light field data to transmit,and the like, where the determination may also include determining anorder, resolution, and the like. For example, for an AR or MR use case,the computing device 402 may selectively create a subset of a lightfield for a scene for the user rather than the entire environment. Assuch, the amount of light field data 410 needed is reduced to only thelight field data relevant to the subset of the scene, where the amountof light field data 410 generated is reduced, the amount of light fielddata 410 transmitted is reduced, and the like. For example, in an ARapplication the real scene may be provided in the background and thelight field data 410 is generated to augment an area of interest, suchas a virtual product placement in an area in the real scene (e.g., auser's home).

In embodiments, light field data 410 may comprise multiple volumes orvolume surfaces of light field data, such as one for a surrounding sceneor environment and one for an object or area of interest. Inembodiments, the multiple volumes or surfaces may have differentresolutions, different sizes, different placements relative to the userdevice, and the like. For instance, a first light field volume may be ofa scene surrounding the user device (e.g., a retail store scene) andhave a relatively lower resolution to that of a second light fieldvolume for an object of interest (e.g., a higher resolution productbeing viewed by a user of the user device). In embodiments, a secondvolume may be located within a first volume. For instance, a firstvolume may have a volume that surrounds the user device such that theuser of the user device is viewing outward into the first volume, suchas where the first volume is a surrounding environment, and a secondvolume is placed inside the first volume where the user device is insidethe first volume but outside the second volume. In this way, a user ofthe user device would be viewing outward into the first volume butinward into the second volume. In an example, a user and user device maybe placed inside the first volume representing a virtual retail storewhere the second volume representing a product is placed inside thefirst volume and next to the user and user device. The user of the userdevice may then look around (i.e. the user device is displaying) andview outward into the surrounding virtual retail store as well asviewing into the second volume to view the product. In this instance,the product, being the object of focus for the user, may be rendered ata higher resolution than the retail store. In embodiments, resolutionmay be dependent on a viewing direction for the user device, such asproviding higher resolution based on the direction the user device ispointing, has pointed to in the past, is predicted to point in the nearfuture, and the like (for example, the light field may have differentresolutions in different directions and different point densities indifferent areas). In embodiments, resolution may be based on whether theuser device is stationary, translating in motion, rotating in motion,moving quickly or slowly, and the like.

Referring to FIGS. 6A-6B and 7A-7B, only the rays of light that areassociated with the area of interest (here the area around the trees)are used to generate light field data 410. So, the light field data 410may only include light field data relevant to the area of interest. Raysof light may also be restricted to include a limited area of interestbut including a present location or view angle (e.g., where the user iscurrently standing and looking) and potentially rays traced from thearea around the location of the user device 412 proximal to a currentlocation (e.g., where the user is likely to look in the future). Thetransmitted light field volume (or surface volume) may be generated byeach potential set of ray tracings for each proximal position of theuser device view. The size of the light field volume may determine theamount of translation and/or rotation that is allowable by the userdevice 412 without the need for additional light field data 410. Inembodiments, the light field volume may be deterministic based on thesize of the available viewable virtual object or environment to the userdevice 412 as well as other considerations, such an order oftransmission for subsequent volumes, resolution of the light field, andthe like.

Referring to FIGS. 6A-6B, the user device 606 may be placed inside thelight field volume, where the light field volume has a light fieldsurface 602 on which a plurality of light field points 604 are located.Referring to FIGS. 7A-7B, it may be seen that each light field datapoint 604 provides a plurality of viewing angles for the user device toview the surrounding environment, such as one view if the user device isat position 606A and a second view if the user device is at position606B. In this way the user device may maintain a view of the surroundingenvironment while moving around within the light field volume (e.g., asdefined by the extent of the light field surface 602). If the userdevice were to move outside the light field volume (e.g., beyond thelight field surface 602) the user device may cease viewing at least aportion of the surrounding scene. As such, it may be desirable tomonitor the movements of the user device with respect to the proximityof the user device 606 to the light field surface 602, such as withrespect to a threshold 702, which may then provide a buffer zone fordetermining whether new light field data should be transmitted orrequested. For instance, a threshold 702 may be set with respect to thelight field surface 602 such that if the user device stays within thethreshold (e.g., doesn't cross the threshold) there may be no need tosend a new light field data to the user device. That is, as long as theuser device is staying well within the light field volume the currentlight field data provided to (for example, stored in) the user devicemay provide views of the surrounding environment without any additionallight field data being transmitted by the computing device. However, ifthe user device 606 crosses the threshold 702 then the computing devicemay transmit additional light field data. In an example, if the userdevice moves from location 606A to location 606C (e.g., staying withinthe threshold) there may be no need to transmit additional light fielddata to the user device. However, if the user device moves from location606A to location 606B (e.g., crossing the threshold) the computingdevice may transmit new light field data and/or the user device mayrequest new light field data. In embodiments, the need to transmitadditional light field data may be determined based on a behavior of theuser device with respect to the threshold (e.g., position, velocity,acceleration with respect to the threshold). In embodiments, thethreshold may be determined by the computing device, the user device, orin combination between the computing device and the user device. In anon-limiting example, the computing device may transmit a thresholdalong with the light field data, where the computing device monitors thelocation of the user device relative to the threshold through monitoringa sensor data sent from the user device to the computing device (e.g., asensor providing location information, velocity information,acceleration information, and the like). In another non-limitingexample, the user device may monitor its location with respect to thethreshold and request new light field data from the computing devicebased on the movement characteristics of the user device (e.g.,velocity, acceleration, and the like). In other embodiments, new lightfield data may be sent and/or requested based on a prediction that theuser device will cross a threshold. In other embodiments, the locationand size of a threshold may be changed based on the behavior of the userdevice (for example, the velocity and acceleration of the user device)and/or the characteristics and status of the connection between the userdevice and the computing device (for example, the threshold may beplaced closer to the device when the device is traveling quickly or theconnection is poor, to allow more of a buffer for providing new lightfield data).

In embodiments, processing of a portion of the light field data 410 maybe provided by the computing device 402, by the user device 412, or incombination. For instance, the computing device 402 may generate thelight field data 410 and then transmit only a portion of the light fielddata 410 generated, or the computing device 402 may generate andtransmit the light field data 410 and provide the user device 412 withlocation information, and then the user device 412, at least in part,processes only a portion of the light field data received based on thelocation information (e.g., location information identifies a locationin the surrounding environment) and then the user device 412 only needsto process changing views for display based on the location information.

Referring to FIG. 8, the computing device 402 may provide the processingfor generating the portion of the light field data, such as in a firststep 810 where the computing device 402 generates the light field data410 representing a portion of a viewed scene by a user device 412, andin a second step 812 the computing device 402 communicates the lightfield data 410 to the user device 412. For instance, previous lightfield data may have indicated that a user of the user device is mostlikely to move in a certain direction while maintain a view in thatdirection. In this instance the computing device 402 may onlycommunicate light field data comprising a light field volume for spacein that direction (e.g., with a certain FOV to allow for the userlooking back and forth somewhat). In embodiments, by limiting thespatial extent of the light field data sent to the user device thecomputing device 402 may be able to increase the volume for the lightfield data (e.g., where the volume is extended from 3 meters to 4meters), refresh the light field data more often (e.g., increasing therate of transmissions of light field data), decrease the processingrequired by the user device, and the like, such as to provide a moreresponsive process to a user action (e.g., when the user begins runningproviding more light field data in the direction of motion), decreasedlatency (e.g., higher refresh rate to the display), and the like.

Referring to FIG. 9, the user device 412 may provide the processing fora portion of the light field data 410, such as in a first step 910 wherethe computing device 402 generates the light field data 410, in a secondstep 912 the computing device 402 communicates the light field data 410and location information, and in a third step 914 the user device 412processes the light field data 410 based on the location information(e.g., where the light field data 410 is made to represent a portion ofthe viewed scene based on the location information). In this instance,rather than the computing device 412 determining and sending a limitedportion of light field data, the computing device 402 may send a full orlarger portion of a light field volume (e.g., a full hemisphere of datarather than say a partial hemisphere) along with location informationthat represents a viewing direction that has a higher probability foruse or a position or subset of interest of the scene. For example, auser may be interacting with an object in a certain location and viewingangle and so the computing device 402 communicates a hemisphere of lightfield data relating to the position of the object. In addition, thecomputing device 402 communicates location information indicating theposition of the object, and with this information the user device may beable to reduce the processing required to render display data. Forinstance, normally the user device may generate display data based onall of the light field data that the computing device 402 communicates,but now the user device may be able to reduce processing based on thelocation information.

In embodiments, the computing device 402 may utilize a prediction of auser device location or behavior in the generation or selection of anext light field data 410 to be communicated to the user device 412. Itmay be possible to predict the future position (e.g., includingorientation of view, where needed) for a user device 412 in a virtual oractual space or where a user of the user device 412 may look in relationto a virtual object, as well as future behavior of a user, such as howthey may interact with a virtual object or a virtual space. Inembodiments, a behavior may be a change in location (e.g., movingforward), a change in viewing perspective (e.g., a shift in viewingangle), a micro-movement (e.g., shaking or shifting), and the like, suchas where the behavior is detected through a sensor or combination ofsensors mounted in the user device or from sensors communicativelycoupled to the user device (e.g., a wearable sensor in communicationwith the user device). The predictions may be based on velocity and/oracceleration vectors of the user, another user, or object, or furtherbased on prior actions of the user (e.g., what the user did in the pastin the space or with the object or in similar spaces or with similarobjects), or further based on other users (e.g., in a virtual gameapplication, when a given character or object enters a space, most usersmove away from the character or object). A prediction may then be usedto determine the content of a next light field data 410 to communicateto the user device 412. Referring to FIG. 10, the computing device 402may utilize prediction information in the generation of light fields,such as in a first step 1010 where the computing device 402 communicatesa first light field data to the user device 412, in a second step 1012where the computing device 402 receives sensor data 406 associated withthe user device 412 (or retrieving behavior data 1002 as stored, such asfrom a previous interaction), in a third step 1014 where the computingdevice 402 predicts a behavior (e.g., a behavior of the user device 412and/or representative of a user behavior) based on the received sensordata 406. Alternatively, the computing device 402 may predict a behaviorbased on other factors, such as other entities or objects in thesurrounding environment. For instance, the predicted behavior may bebased on an object entering the scene. In a fourth step 1016 thecomputing device 402 generates a second light field data based on thepredicted behavior. Alternately, the second light field may not begenerated but rather selected based on the predicted behavior. Forinstance, light field data may already have been generated and theprediction determines which light field data to communicate to the userdevice 412. In a fifth step 1018 the computing device 402 communicatesthe second light field to the user device 412.

In embodiments, a prediction may assist with determining what lightfield data 410 to send to the user device 412 next, to determine thesize of light field volumes to send or the order of a series of volumes(e.g., based on the likely order of positions of the user, such assending a bigger volume or box if the user's head is moving fast and asmaller box when they are perceived as being stationary or standingrelatively still).

In embodiments, a prediction may assist with determining the resolution(or quality) of the light field data 410. For example, the computingdevice 402 may provide higher resolution light field data 410 for anarea around objects a user is likely to examine but provide lowerresolution data (or possibly no data or zero resolution) for areasaround objects a user is less likely to view. Prediction may impact thecreation of the light field data 410 (e.g., do not create or create onlylower resolution light field data for areas users are unlikely to viewor will only view when moving quickly so only lower resolution isneeded) or may impact transmission (e.g., the light field data alreadyexists at a higher resolution than needed, so reduce the resolution andthen transmit).

In embodiments, a prediction may enable a volume optimization, such asby not generating a full volume like a cylinder, a sphere, or a cube,but instead generating a volume that is more directional such as in ahalf of a cube or sphere relative to the direction a user is or ispredicted to be facing or moving or where the volume contains more datain certain regions or with respect to views in certain directions thanothers. In this instance, if the user were to turn around or move inother than a predicted direction, they may see blackness and the extentof the light field would become apparent. However, since the computingdevice 402 is likely to be communicating many light fields per second itmay keep up with the head motions and orientation of the user. Inembodiments, this may become an issue if the network connection wasinterrupted and the user turned around.

In embodiments, a prediction may be used to set a density of light fieldpoints, where the density of the light field points may not be requiredto be uniform. For instance, this may enable a high density of points ona light field surface volume near predicted viewing vectors and a lowerdensity of points in places such as behind the user if a fully enclosedvolume is not provided.

In embodiments, a prediction may be used to set a FOV in combinationwith setting a resolution. For instance, if the head of a user is movingthen the computing device 402 may translate the FOV towards thedirection of motion and send light field data 410 adjacent to thecurrent light field and adjust resolution to be higher in thatdirection.

In embodiments, a prediction may assist in generating light field data410 for multiple head or viewing positions against a probabilitygradient of possible head or other positions based on the previous heador other position and any translational vector for the head or otherposition. For instance, probabilistic interpolation may be used toreduce the number of rays being transmitted proportional to theprobabilistic location of a head or other position. In embodiments, thecomputing device 402 may address the network interruptions or jitter byallowing the virtual environment to allow for translational updateswithout an environmental update.

In embodiments, a prediction may be enhanced through machine learning topredict the need for light field data 410 as a user interacts with theenvironment scene or object(s), such as through the use of real timelight tracing technology.

In embodiments, a prediction may be associated with a user being near,holding, or in contact with on object, such as in association withrendering or raytracing with respect to objects the user is holding ontoor touching (such as through the user device 412). That is, the use oflight fields may not only help to solve latency associated with themotion of the user or user device, but also with respect to hand or limbmotion, to prevent or reduce latency with an object they are interactingwith (e.g., the object doesn't move smoothly with a user's hand motion).Hand motion latency may not make the user sick, but it may reduce theuser experience of immersion. In embodiments, the computer device mayoptimize for rending translations and rotations of the user device 412with respect to objects the user is directly manipulating through theuser device 412 to produce a low perceptual latency. For example, thecomputing device could be configured to predict (e.g. based on the userdevice location or field of view) the user is likely to interact with anobject and on that basis, transmit a volume of light field datasurrounding that object that is sufficiently large to accommodate handtranslations or rotations of the object. In embodiments, the computingdevice may generate a light field for the volume surrounding the object,where the user device renders the object locally. In embodiments, thecomputing device 402 may provide object related light field datageneration through (parallel) processing of light fields for multipleobjects and performance requirements, stitching of multiple light fieldsinto a single light field for interactions, and the like. For instance,the computing device 402 may maintain parallel processing for thesurrounding scene and for a particular object. In other embodiments, thecomputing device may instead generate and transmit to the user device alight field volume for the volume environment or scene surrounding theobject (i.e. a light field volume sufficiently large to exclude theobject), while the user device is configured to render the objectlocally (e.g. by its own means).

As described herein, a light field is a vector function that describesthe amount of light flowing in every direction through every point inspace in a volume of space, such as in a data model for storing thelight environment in that volume of space. The space of all possiblelight rays is given by the five-dimensional plenoptic function, and themagnitude of each ray is given by the radiance. With respect togeometric optics (e.g., to incoherent light and to objects larger thanthe wavelength of light) the fundamental carrier of light is a ray oflight, such as depicted through ray tracking (such as depicted in FIGS.6 and 7). The measure for the amount of light traveling along a ray isradiance, for example measured in watts per steradian per meter squared(m2). The steradian is a measure of solid angle, where 4π steradiansrepresents a ‘all view’ volume (e.g., where an all view light fieldsurface volume would provide visual information for all directions fromthe user/user device). The radiance along all light rays in a region ofthree-dimensional space illuminated by an unchanging arrangement oflights is called the plenoptic function. The plenoptic illuminationfunction is an idealized function used in computer vision and computergraphics to express the image of a scene from any possible viewingposition at any viewing angle at any point in time. Since rays in spacemay be parameterized by three coordinates, x, y, and z and two angles θand ϕ, the plenoptic function is a five-dimensional function. However,the plenoptic function contains redundant information, because theradiance along a ray remains constant from point to point along itslength. After removal of the redundant information the light fieldbecomes a four-dimensional function. Light fields may be generated byrendering image data or photographing a real scene (e.g., using amicro-lens array and interpolation between cameras/lens), producinglight field views for a large number of viewpoints. The number andarrangement of images in a light field along with the resolution of eachimage may determine the sampling of the 4D light field. In embodiments,light fields may be stored in a file, such as with a ‘.lif’ file format,which may be a single file to describe a light field, including a headersection (e.g., with information regarding the number of slabs andspatial geometry) and a binary section (with the light field data).

In embodiments, as described herein, the computing device 402 may setthe resolution or quality of a light field when communicating lightfield data 410, such as varying the amount of data on a surface volume.In theory there are an infinite number of rays that travel through apoint in the light field (e.g., at a maximum resolution), then for lowerresolution the computing device 402 may limit the number of rays, suchas to, for example, 1000 of those rays and for even lower resolution to,say, 100 of the rays. In embodiments, this reduction in resolution maybe applied to certain directions more than others, such as in setting adirectional preference for high resolution. For example, a higherresolution may be set for the direction the user device 412 is facing oris predicted to face and lower to the sides and further lower behind theview of the user device 412. In embodiments, another resolutionparameter may be the number of points per surface area (e.g., lightfield point density for a given surface volume) or included in a lightfield volume in the light field. In this way a light field point densitymay be specified and adjusted as a resolution parameter. As such, therays per point may be optimized with respect to density, which may alsobe adjusted with respect to a location parameter, such as viewingdirection (e.g., again, setting a higher density for directions the userdevice 412 is currently facing or anticipated to face). Further, theresolution may be specified for a portion of a view, such as in adirectional optimization. As shown in FIG. 6, directional optimizationmay be applied to the light field based on a viewing perspective of theuser device 412 because the user device 412 is only viewing a point fromone of two sides of an object, where the generation of light field data410 for the user device 412 may only need to be concerned with the raysof light leaving a subject of view and coming to the user device 412,not the rays in the direction to the subject. That is, generation of thelight field data 410 may need to be focus on rays of light entering thevolume, not leaving it.

In embodiments, resolution adjustments may be implemented throughdown-sampling or selection of traced rays in a light field to have alower resolution light field than the infinite rays and infinite pointsthat exists in the real world. However, if the number of light raysand/or the number of light field points is such that it is below thetheoretical maximum (i.e. infinity), gaps may possibly exist between therays and/or between the points for which blending and/or interpolationmay be needed. In embodiments, blending and/or interpolation may beperformed when the light field is generated (e.g., utilizing an opticaltransistor), when the light field is visualized (e.g., utilizing anoptical receiver), and the like. In embodiments, dynamic resolutionstitches may be present in the same light field, thus allowing for lowerresolution light fields to exist in the same FOV as higher resolutionlight fields.

Referring to FIGS. 4, 5 and 8-10, the computing device 402, inassociation with the image processor 408, may provide a benefit to theuser experience of viewing interactive visual content being delivered tothe user device 412, where light field data 410 is provided to the userdevice 412 rather than transmitting video stream data and the like. Thisbenefit is realized at least in part because light field data 410 may beprocessed more quickly to provide a display view to the user device 412than if using other methods. For instance, a virtual reality user deviceneeds only the light field data 410 to create the image for the display520 of the user device 412, such as based on the position and directionof view of the user device 412. For a given position, the computingdevice 402 may compute and provide data associated with a light fieldsurface volume of light field data 410, such as described by a grid oflight field points and for every light field point in that grid (or anapproximation of every point, such as though interpolation) determininga resolution for the number of light field points and how every ray oflight goes through that light field point. In addition, the computingdevice 402 may set a volumetric size for each user device position(e.g., plus or minus 1 meter) so that there will be light field data 410for positions around the user device location The result is a(streaming) transmission of a series of light field volumes of lightfield data 410 over time to the user device 412 corresponding todifferent positions of the user device 412. Thus, in some embodiments,the computing device 402 does not have to take into account the angle ordirection of view of the user device 412 when generating light fielddata 410 since all needed angles and directions (as prioritized throughdirectional preference) may be included in the light field data 410 fora given position of the user device 412. This in turn may improve theuser experience by reducing the need for transmitted updates to the userdevice because the light field data 410 received by the user devicecontains information for a plurality of views, such as for movementswithin the light field volume.

In embodiments, light field data 410 may be generated with respect toimage content, such as captured from a camera or series of cameras instill or video format, from a pre-recorded video file or from areal-time camera, and the like. Image content may be used to generatelight field data 410 from a single view, multiple views, a full 360degree view, a full 4π steradian view, and the like, in a single lightfield surface volume, multiple light field surface volumes, and thelike. In embodiments, real-time image data may be captured by a cameraor series of cameras and included in sensor data 406 to be used by thecomputing device 402 to generate light field data 410, such as where thecamera(s) is mounted on the user device 412, communicatively coupled tothe user device 412 (e.g., located proximate the user device), and thelike. For example, a camera may be integrated into a VR, AR, or MR userdevice for viewing a surrounding environment, where the camera sends theimage content from the camera to the computing device 402 for use ingenerating light field data 410 for use in a VR, AR, or MR interactivevideo experience. In embodiments, the camera my capture a partial (e.g.,single or multiple camera) or full view (e.g., array of cameras) of thesurrounding environment, enabling the computing device 402 to generate acorresponding light field data 410. For instance, in a full 360 degreecamera view the computing device 402 may generate a full 360 degreelight field view for communicating to the user device 412, permittinguser device 412 to rotate around and display the surrounding scene asdisplayed through the light field data 410 without the computing device402 needing to send additional light field data 410. In another example,a series or array of cameras located in the surrounding environment butseparate from the user device 412 may be used to generate image contentfor communicating to the computing device 402 for generation of thelight field data 410, such as where the image content is transferred tothe computing device 402 through the user device 412, transferred to thecomputing device 402 through a network connection, and the like.

In embodiments, light field data may be generated from ray tracing basedon a 3D environment or real-time video image content may be capturedfrom camera(s) in one environment in order to generate light field data410 for communication to a user device 412 in a second environment toprovide a user of the user device 412 with a view of the firstenvironment. For instance, a customer may wear a VR headset in order tobe fully immersed in a remote store environment to replicate a physicalinstore experience, such as to place a user in a store where user caninteract with sales people, and the like, take into account changes inthe store, actions of sales people, and the like, in a dynamic real-timeexperience. In another instance, a customer may wear an AR or MR deviceto have a salesperson virtually ‘enter the home’ of the user, such asshowing the user products and product demonstrations. In embodiments,the user and the sales person may utilize wearable user devices tointeract together. In embodiments, the user device 412 may be a handheld smart phone or other mobile device that provides a virtual ‘window’to the second environment, such as where the user experiences aninteractive viewing window into a store through the smart phone display,and where sensors mounted on the smart phone, on the user, or in theuser's environment provide sensor data 406 to the computing device 402for generating light field data 410.

In embodiments, light field data 410 may be used to simulate a productinteraction. For instance, a customer using a user device 412 may beable to interact with a product, such as either a real view of a product(e.g., in a store) or with a virtual product (e.g., computer generated).For example, light field data 410 may be generated for a product withhigh resolution for the product and low resolution for the areasurrounding the product, high resolution for areas of the product a useris viewing or where a user is in contact with the product, and the like.In an example, the user may be wearing an AR or MR user device and viewthe product as an overlay onto the user's surroundings, where lightfield data 410 is dynamically generated to adjust resolution to theactions of the user as viewed through a camera on the user device 412(e.g., the user device camera viewing the user's hand reaching out tothe product and the light field resolution increasing for product areanear the user's hand). In another example, the user device 412 may be asmart phone or other mobile device adapted to operate in an MR mode(e.g., through an application on the phone), and the light field dataresolution dynamically increases as the user moves the phone ‘closer tothe product’ in a virtual interaction with the product, where sensors inthe smart phone send position and/or motion sensor data to the computingdevice 402 (e.g., along with any camera generated image content for anyportion of the user viewed by the user device camera, such as where theuser moves the phone forward to get closer to the product and alsoreaches in with a hand near the product—all with views changing as thesmart phone translates and rotates).

In embodiments, light field data 410 may be used to provide a virtualservice experience, where light field data 410 provides an improvedreal-time user experience through a more continuous immersive virtualexperience. For instance, a user may virtually visit a museum in aforeign country through an AR, VR, or MR user device, where the uservirtually walks through the museum, viewing the surroundings andexhibits, interacting with museum personnel and visitors, and the like,where light field data 410 is generated based on behavior (e.g.,behavior of the user as viewed through a camera, behavior of the userdevice 412 a detected through position and/or motion sensor data, andthe like), viewing FOV (e.g., adjusting the size of a light field volumeand/or resolution based on a detected motion of the user device),interaction with objects and/or people (e.g., increasing resolution whena user is viewed through the user device camera to be coming near anobject or area of interest), and the like.

In embodiments, light field data 410 may be used to view products and/orservices at the location of the user device 412. For instance, with anAR or MR user device products, people, places, and the like, may bevirtually brought into the home of a user, such as viewing products in alocation where the product would go in the home (e.g., see a chair in aliving room or a vase on a table), having a sales person provide aservice in the home (e.g., teach a class on how to build a table in auser's workshop and the like), where the use of light field data 410provides a consistent immersive user experience. In an example, a usermay wear a head-worn AR user device to view an overlay of a new virtualcouch on the background of the actual living room through light fieldrendering, allowing the user of the device to walk around the livingroom and view the virtual couch from any angle, seeing how it looks fromdifferent perspectives in the living room, and where the computingdevice 402 adjusts the light field data 410 based on the movements andperspective view of the user device 412. For instance, a user may benddown and look carefully at a back portion of the couch and the computingdevice 402 increases resolution around the area of interest on thecouch. In this instance, because the user device 412 has a lowprobability for turning around in a direction opposite the couch, thecomputing device 402 may not provide light field data 410 behind thecurrent view of the user device 412 (e.g., probability is high that theuser will stay focused on the couch and not the rest of the room). Inembodiments, a second user (e.g., a spouse) may operate a second userdevice 412 and the two user's devices may interact. Further, thecomputing device 402 may take advantage of overlapping views for the twouser devices when generating the sensor data 406 for the two userdevices.

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software, program codes,and/or instructions on a processor. The processor may be part of aserver, cloud server, client, network infrastructure, mobile computingplatform, stationary computing platform, or other computing platform. Aprocessor may be any kind of computational or processing device capableof executing program instructions, codes, binary instructions and thelike. The processor may be or include a signal processor, digitalprocessor, embedded processor, microprocessor or any variant such as aco-processor (math co-processor, graphic co-processor, communicationco-processor and the like) and the like that may directly or indirectlyfacilitate execution of program code or program instructions storedthereon. In addition, the processor may enable execution of multipleprograms, threads, and codes. The threads may be executed simultaneouslyto enhance the performance of the processor and to facilitatesimultaneous operations of the application. By way of implementation,methods, program codes, program instructions and the like describedherein may be implemented in one or more thread. The thread may spawnother threads that may have assigned priorities associated with them;the processor may execute these threads based on priority or any otherorder based on instructions provided in the program code. The processormay include memory that stores methods, codes, instructions and programsas described herein and elsewhere. The processor may access a storagemedium through an interface that may store methods, codes, andinstructions as described herein and elsewhere. The storage mediumassociated with the processor for storing methods, programs, codes,program instructions or other type of instructions capable of beingexecuted by the computing or processing device may include but may notbe limited to one or more of a CD-ROM, DVD, memory, hard disk, flashdrive, RAM, ROM, cache and the like.

A processor may include one or more cores that may enhance speed andperformance of a multiprocessor. In embodiments, the process may be adual core processor, quad core processors, other chip-levelmultiprocessor and the like that combine two or more independent cores(called a die).

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software on a server,cloud server, client, firewall, gateway, hub, router, or other suchcomputer and/or networking hardware. The software program may beassociated with a server that may include a file server, print server,domain server, internet server, intranet server and other variants suchas secondary server, host server, distributed server and the like. Theserver may include one or more of memories, processors, computerreadable media, storage media, ports (physical and virtual),communication devices, and interfaces capable of accessing otherservers, clients, machines, and devices through a wired or a wirelessmedium, and the like. The methods, programs or codes as described hereinand elsewhere may be executed by the server. In addition, other devicesrequired for execution of methods as described in this application maybe considered as a part of the infrastructure associated with theserver.

The server may provide an interface to other devices including, withoutlimitation, clients, other servers, printers, database servers, printservers, file servers, communication servers, distributed servers andthe like. Additionally, this coupling and/or connection may facilitateremote execution of program across the network. The networking of someor all of these devices may facilitate parallel processing of a programor method at one or more location without deviating from the scope ofthe disclosure. In addition, any of the devices attached to the serverthrough an interface may include at least one storage medium capable ofstoring methods, programs, code and/or instructions. A centralrepository may provide program instructions to be executed on differentdevices. In this implementation, the remote repository may act as astorage medium for program code, instructions, and programs.

The software program may be associated with a client that may include afile client, print client, domain client, internet client, intranetclient and other variants such as secondary client, host client,distributed client and the like. The client may include one or more ofmemories, processors, computer readable media, storage media, ports(physical and virtual), communication devices, and interfaces capable ofaccessing other clients, servers, machines, and devices through a wiredor a wireless medium, and the like. The methods, programs or codes asdescribed herein and elsewhere may be executed by the client. Inaddition, other devices required for execution of methods as describedin this application may be considered as a part of the infrastructureassociated with the client.

The client may provide an interface to other devices including, withoutlimitation, servers, other clients, printers, database servers, printservers, file servers, communication servers, distributed servers andthe like. Additionally, this coupling and/or connection may facilitateremote execution of program across the network. The networking of someor all of these devices may facilitate parallel processing of a programor method at one or more location without deviating from the scope ofthe disclosure. In addition, any of the devices attached to the clientthrough an interface may include at least one storage medium capable ofstoring methods, programs, applications, code and/or instructions. Acentral repository may provide program instructions to be executed ondifferent devices. In this implementation, the remote repository may actas a storage medium for program code, instructions, and programs.

The methods and systems described herein may be deployed in part or inwhole through network infrastructures. The network infrastructure mayinclude elements such as computing devices, servers, routers, hubs,firewalls, clients, personal computers, communication devices, routingdevices and other active and passive devices, modules and/or componentsas known in the art. The computing and/or non-computing device(s)associated with the network infrastructure may include, apart from othercomponents, a storage medium such as flash memory, buffer, stack, RAM,ROM and the like. The processes, methods, program codes, instructionsdescribed herein and elsewhere may be executed by one or more of thenetwork infrastructural elements.

The methods, program codes, and instructions described herein andelsewhere may be implemented in different devices which may operate inwired or wireless networks. Examples of wireless networks include 4^(th)Generation (4G) networks (e.g. Long Term Evolution (LTE)) or 5^(th)Generation (5G) networks, as well as non-cellular networks such asWireless Local Area Networks (WLANs). However, the principles describedtherein may equally apply to other types of networks.

The operations, methods, programs codes, and instructions describedherein and elsewhere may be implemented on or through mobile devices.The mobile devices may include navigation devices, cell phones, mobilephones, mobile personal digital assistants, laptops, palmtops, netbooks,pagers, electronic books readers, music players and the like. Thesedevices may include, apart from other components, a storage medium suchas a flash memory, buffer, RAM, ROM and one or more computing devices.The computing devices associated with mobile devices may be enabled toexecute program codes, methods, and instructions stored thereon.Alternatively, the mobile devices may be configured to executeinstructions in collaboration with other devices. The mobile devices maycommunicate with base stations interfaced with servers and configured toexecute program codes. The mobile devices may communicate on a peer topeer network, mesh network, or other communications network. The programcode may be stored on the storage medium associated with the server andexecuted by a computing device embedded within the server. The basestation may include a computing device and a storage medium. The storagedevice may store program codes and instructions executed by thecomputing devices associated with the base station.

The computer software, program codes, and/or instructions may be storedand/or accessed on machine readable media that may include: computercomponents, devices, and recording media that retain digital data usedfor computing for some interval of time; semiconductor storage known asrandom access memory (RAM); mass storage typically for more permanentstorage, such as optical discs, forms of magnetic storage like harddisks, tapes, drums, cards and other types; processor registers, cachememory, volatile memory, non-volatile memory; optical storage such asCD, DVD; removable media such as flash memory (e.g. USB sticks or keys),floppy disks, magnetic tape, paper tape, punch cards, standalone RAMdisks, Zip drives, removable mass storage, off-line, and the like; othercomputer memory such as dynamic memory, static memory, read/writestorage, mutable storage, read only, random access, sequential access,location addressable, file addressable, content addressable, networkattached storage, storage area network, bar codes, magnetic ink, and thelike.

The methods and systems described herein may transform physical and/oror intangible items from one state to another. The methods and systemsdescribed herein may also transform data representing physical and/orintangible items from one state to another, such as from usage data to anormalized usage dataset.

The elements described and depicted herein, including in flow charts andblock diagrams throughout the figures, imply logical boundaries betweenthe elements. However, according to software or hardware engineeringpractices, the depicted elements and the functions thereof may beimplemented on machines through computer executable media having aprocessor capable of executing program instructions stored thereon as amonolithic software structure, as standalone software modules, or asmodules that employ external routines, code, services, and so forth, orany combination of these, and all such implementations may be within thescope of the present disclosure. Examples of such machines may include,but may not be limited to, personal digital assistants, laptops,personal computers, mobile phones, other handheld computing devices,medical equipment, wired or wireless communication devices, transducers,chips, calculators, satellites, tablet PCs, electronic books, gadgets,electronic devices, devices having artificial intelligence, computingdevices, networking equipment, servers, routers and the like.Furthermore, the elements depicted in the flow chart and block diagramsor any other logical component may be implemented on a machine capableof executing program instructions. Thus, while the foregoing drawingsand descriptions set forth functional aspects of the disclosed systems,no particular arrangement of software for implementing these functionalaspects should be inferred from these descriptions unless explicitlystated or otherwise clear from the context. Similarly, it will beappreciated that the various steps identified and described above may bevaried, and that the order of steps may be adapted to particularapplications of the techniques disclosed herein. All such variations andmodifications are intended to fall within the scope of this disclosure.As such, the depiction and/or description of an order for various stepsshould not be understood to require a particular order of execution forthose steps, unless required by a particular application, or explicitlystated or otherwise clear from the context.

The methods and/or processes described above, and steps thereof, may berealized in hardware, software or any combination of hardware andsoftware suitable for a particular application. The hardware may includea general-purpose computer and/or dedicated computing device or specificcomputing device or particular aspect or component of a specificcomputing device. The processes may be realized in one or moremicroprocessors, microcontrollers, embedded microcontrollers,programmable digital signal processors or other programmable device,along with internal and/or external memory. The processes may also, orinstead, be embodied in an application specific integrated circuit, aprogrammable gate array, programmable array logic, or any other deviceor combination of devices that may be configured to process electronicsignals. It will further be appreciated that one or more of theprocesses may be realized as a computer executable code capable of beingexecuted on a machine readable medium.

The computer executable code may be created using a structuredprogramming language such as C, an object oriented programming languagesuch as C++, or any other high-level or low-level programming language(including assembly languages, hardware description languages, anddatabase programming languages and technologies) that may be stored,compiled or interpreted to run on one of the above devices, as well asheterogeneous combinations of processors, processor architectures, orcombinations of different hardware and software, or any other machinecapable of executing program instructions.

Thus, in one aspect, each method described above, and combinationsthereof may be embodied in computer executable code that, when executingon one or more computing devices, performs the steps thereof. In anotheraspect, the methods may be embodied in systems that perform the stepsthereof and may be distributed across devices in a number of ways, orall of the functionality may be integrated into a dedicated, standalonedevice or other hardware. In another aspect, the means for performingthe steps associated with the processes described above may include anyof the hardware and/or software described above. All such permutationsand combinations are intended to fall within the scope of the presentdisclosure.

What is claimed is:
 1. A computer-implemented method comprising:communicating, by a computing device, a first light field data to a userdevice, wherein the first light field data is limited to a first volumeof space that represents a first plurality of views for display on theuser device; receiving, by the computing device, a sensor dataassociated with the user device; predicting a behavior based at least inpart from the received sensor data; generating with the computing devicea second light field data based at least in part on the predictedbehavior, wherein the second light field data is limited to a secondvolume of space that is different from the first volume of space andrepresents a second plurality of views for display on the user device;and communicating, by the computing device, the second light field datato the user device.
 2. The method of claim 1, wherein the sensor data isgenerated by at least one sensor integrated into the user device and thebehavior is a behavior of the user device.
 3. The method of claim 2,wherein the sensor data is at least in part translational sensor dataand the predicted behavior is a predicted future position of the userdevice located within the second volume of space.
 4. The method of claim2, wherein the sensor data is at least in part rotational sensor dataand the predicted behavior is a predicted future viewing angle of theuser device within the second volume of space.
 5. The method of claim 2,wherein the predicted behavior is based on sensor data stored from aprior action of the user device while the user device was previously inthe first volume of space.
 6. A computer-implemented method comprising:receiving, by a computing device associated with an image processor, asensor data associated with a user device, wherein the sensor data is inresponse to a first light field data limited to a first volume of spacethat represents a first plurality of views for display on the userdevice; generating, by the computing device, a second light field databased on a predicted behavior determined at least in part by thereceived sensor data, wherein the second light field data is limited toa second volume of space that is different from the first volume ofspace and represents a second plurality of views for display on the userdevice; and communicating, by the computing device, the second lightfield data to the user device.
 7. The method of claim 6, wherein thefirst light field data and the second light field data each comprise athree-dimensional volume describing the light flowing in a plurality ofdirections through a plurality of light field data points.
 8. The methodof claim 6, wherein the sensor data is at least in part translationalsensor data and the predicted behavior is a predicted future position ofthe user device located within the second volume of space.
 9. The methodof claim 6, wherein the sensor data is at least in part rotationalsensor data and the predicted behavior is a predicted future viewingangle of the user device within the second volume of space.
 10. Themethod of claim 6, wherein the predicted behavior is based on sensordata stored from a prior action of the user device while the user devicewas previously in the first volume of space.
 11. The method of claim 6,wherein the predicted behavior is based on an action of a second userdevice while the user device was in the first volume of space.
 12. Themethod of claim 6, wherein the first volume of space is proximate theuser device and the second volume of space is determined by thepredicted behavior.
 13. The method of claim 6, wherein the second lightfield data comprises a light field resolution and the light fieldresolution is determined by the predicted behavior.
 14. The method ofclaim 6, wherein the second light field data comprises a directionalitywith respect to a user device perspective and the directionality isdetermined by the predicted behavior.
 15. The method of claim 6, whereinthe second light field data comprises a light field point density andthe light field point density is determined by the predicted behavior.16. The method of claim 6, wherein the user device is a head-worndevice, and the predicted behavior is associated with a head position ofthe user device.
 17. The method of claim 6, wherein the predictedbehavior is determined at least in part through machine learningassociated with previous behavior.
 18. The method of claim 6, whereinthe predicted behavior is determined to be associated with a usercontact with an object.
 19. The method of claim 6, wherein the userdevice is communicatively coupled to a display for presentation of viewsfor display on the user device.
 20. The method of claim 6, wherein thesensor data comprises motion sensor data.
 21. A system comprising: acomputing device associated with an image processor, the computingdevice configured to store a set of instructions that, when executed,cause the computing device to: receive, by the computing deviceassociated with the image processor, a sensor data associated with auser device, wherein the sensor data is in response to a first lightfield data limited to a first volume of space that represents a firstplurality of views for display on the user device; generate, by thecomputing device, a second light field data based on a predictedbehavior determined at least in part by the received sensor data,wherein the second light field data is limited to a second volume ofspace that is different from the first volume of space and represents asecond plurality of views for display on the user device; andcommunicate, by the computing device, the second light field data to theuser device.
 22. The system of claim 21, wherein the sensor data isgenerated by at least one sensor integrated into the user device and thepredicted behavior is a behavior of the user device.
 23. The system ofclaim 22, wherein the sensor data is at least in part translationalsensor data and the predicted behavior is a predicted future position ofthe user device located within the second volume of space.
 24. Thesystem of claim 22, wherein the sensor data is at least in partrotational sensor data and the predicted behavior is a predicted futureviewing angle of the user device within the second volume of space. 25.The system of claim 22, wherein the predicted behavior is based onsensor data stored from a prior action of the user device while the userdevice was previously in the first volume of space.